[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 205 - Failure

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 205 - Failure

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/205/

2 tests failed.
REGRESSION:  org.apache.lucene.search.TestDocTermOrdsRewriteMethod.testRegexps

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
        at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestDocTermOrdsRewriteMethod

Error Message:
Suite timeout exceeded (>= 7200000 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
        at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)




Build Log:
[...truncated 1605 lines...]
[junit4:junit4] Suite: org.apache.lucene.search.TestDocTermOrdsRewriteMethod
[junit4:junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsRewriteMethod -Dtests.method=testRegexps -Dtests.seed=2923E24EEDCF7E1F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Etc/GMT -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   7200s J0 | TestDocTermOrdsRewriteMethod.testRegexps <<<
[junit4:junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=tr, timezone=Etc/GMT
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=159567728,total=266862592
[junit4:junit4]   2> NOTE: All tests run in this JVM: [TestSearchForDuplicates, TestSegmentTermDocs, TestRegexpRandom2, InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, TestIndexWriterConfig, TestBlockPostingsFormat3, TestSearchAfter, TestNumericRangeQuery32, TestCopyBytes, TestDateTools, TestPayloadTermQuery, TestPriorityQueue, TestExternalCodecs, TestRamUsageEstimator, TestNoMergeScheduler, TestFieldCacheTermsFilter, Nested1, TestCachingCollector, TestRecyclingByteBlockAllocator, TestNumericRangeQuery64, TestPrefixInBooleanQuery, TestTimeLimitingCollector, TestSimpleAttributeImpl, TestSimpleSearchEquivalence, TestDocumentWriter, TestLucene40PostingsReader, TestWildcardRandom, TestIndexableField, TestSort, TestAllFilesHaveCodecHeader, TestLevenshteinAutomata, TestRandomStoredFields, TestDoubleBarrelLRUCache, TestSpansAdvanced2, TestSimilarityBase, TestSloppyPhraseQuery2, TestMultiTermQueryRewrites, TestIndexWriterUnicode, TestRollingUpdates, TestTwoPhaseCommitTool, TestIndexInput, TestBytesRef, TestSimpleExplanations, TestRegexpQuery, Test2BDocs, TestForTooMuchCloning, TestBagOfPostings, TestCollectionUtil, TestMultiValuedNumericRangeQuery, TestToken, TestParallelReaderEmptyIndex, TestLazyProxSkipping, TestStressIndexing, TestCustomNorms, TestShardSearching, TestMockCharFilter, TestCompiledAutomaton, TestSubScorerFreqs, TestPersistentSnapshotDeletionPolicy, TestLookaheadTokenFilter, TestPrefixFilter, TestDocValuesWithThreads, TestRamUsageEstimatorOnWildAnimals, TestSurrogates, TestLucene42DocValuesFormat, TestLucene40DocValuesFormat, Nested1, Nested2, Nested3, TestComplexExplanationsOfNonMatches, TestBlockPostingsFormat2, TestIndexWriterMerging, Test2BPositions, TestSegmentReader, TestWildcard, TestConstantScoreQuery, TestSpanExplanationsOfNonMatches, TestFilterIterator, TestSloppyPhraseQuery, TestTermsEnum2, Nested1, TestTopScoreDocCollector, TestForceMergeForever, TestIndexWriter, TestScoreCachingWrappingScorer, Nested1, TestSetOnce, TestDemo, TestTermRangeQuery, TestDocValuesFormat, TestIndexWriterReader, TestBytesStore, TestIndexWriterExceptions, TestDeletionPolicy, TestBooleanOr, TestBasics, TestFlushByRamOrCountsPolicy, TestIndexWriterDelete, TestNRTThreads, TestStressAdvance, TestConcurrentMergeScheduler, TestHighCompressionMode, TestDocumentsWriterStallControl, TestTopDocsMerge, TestCompoundFile, TestLongPostings, TestMinimize, TestDocTermOrdsRewriteMethod]
[junit4:junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsRewriteMethod -Dtests.seed=2923E24EEDCF7E1F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Etc/GMT -Dtests.file.encoding=ISO8859-1
[junit4:junit4] ERROR   0.00s J0 | TestDocTermOrdsRewriteMethod (suite) <<<
[junit4:junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)
[junit4:junit4] Completed on J0 in 7223.18s, 2 tests, 2 errors <<< FAILURES!

[...truncated 11 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:388: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:50: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:877: There were test failures: 351 suites, 2183 tests, 1 suite-level error, 1 error, 29 ignored (17 assumptions)

Total time: 144 minutes 26 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 205 - Failure

Robert Muir
NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity,
locale=tr, timezone=Etc/GMT

I think surrogates dance + nightly caused this to be super slow.

On Mon, Mar 11, 2013 at 3:18 PM, Apache Jenkins Server
<[hidden email]> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/205/
>
> 2 tests failed.
> REGRESSION:  org.apache.lucene.search.TestDocTermOrdsRewriteMethod.testRegexps
>
> Error Message:
> Test abandoned because suite timeout was reached.
>
> Stack Trace:
> java.lang.Exception: Test abandoned because suite timeout was reached.
>         at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)
>
>
> FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestDocTermOrdsRewriteMethod
>
> Error Message:
> Suite timeout exceeded (>= 7200000 msec).
>
> Stack Trace:
> java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
>         at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)
>
>
>
>
> Build Log:
> [...truncated 1605 lines...]
> [junit4:junit4] Suite: org.apache.lucene.search.TestDocTermOrdsRewriteMethod
> [junit4:junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsRewriteMethod -Dtests.method=testRegexps -Dtests.seed=2923E24EEDCF7E1F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Etc/GMT -Dtests.file.encoding=ISO8859-1
> [junit4:junit4] ERROR   7200s J0 | TestDocTermOrdsRewriteMethod.testRegexps <<<
> [junit4:junit4]    > Throwable #1: java.lang.Exception: Test abandoned because suite timeout was reached.
> [junit4:junit4]    >    at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)
> [junit4:junit4]   2> NOTE: test params are: codec=Lucene3x, sim=DefaultSimilarity, locale=tr, timezone=Etc/GMT
> [junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=2,free=159567728,total=266862592
> [junit4:junit4]   2> NOTE: All tests run in this JVM: [TestSearchForDuplicates, TestSegmentTermDocs, TestRegexpRandom2, InBeforeClass, InAfterClass, InTestMethod, NonStringProperties, TestIndexWriterConfig, TestBlockPostingsFormat3, TestSearchAfter, TestNumericRangeQuery32, TestCopyBytes, TestDateTools, TestPayloadTermQuery, TestPriorityQueue, TestExternalCodecs, TestRamUsageEstimator, TestNoMergeScheduler, TestFieldCacheTermsFilter, Nested1, TestCachingCollector, TestRecyclingByteBlockAllocator, TestNumericRangeQuery64, TestPrefixInBooleanQuery, TestTimeLimitingCollector, TestSimpleAttributeImpl, TestSimpleSearchEquivalence, TestDocumentWriter, TestLucene40PostingsReader, TestWildcardRandom, TestIndexableField, TestSort, TestAllFilesHaveCodecHeader, TestLevenshteinAutomata, TestRandomStoredFields, TestDoubleBarrelLRUCache, TestSpansAdvanced2, TestSimilarityBase, TestSloppyPhraseQuery2, TestMultiTermQueryRewrites, TestIndexWriterUnicode, TestRollingUpdates, TestTwoPhaseCommitTool, TestIndexInput, TestBytesRef, TestSimpleExplanations, TestRegexpQuery, Test2BDocs, TestForTooMuchCloning, TestBagOfPostings, TestCollectionUtil, TestMultiValuedNumericRangeQuery, TestToken, TestParallelReaderEmptyIndex, TestLazyProxSkipping, TestStressIndexing, TestCustomNorms, TestShardSearching, TestMockCharFilter, TestCompiledAutomaton, TestSubScorerFreqs, TestPersistentSnapshotDeletionPolicy, TestLookaheadTokenFilter, TestPrefixFilter, TestDocValuesWithThreads, TestRamUsageEstimatorOnWildAnimals, TestSurrogates, TestLucene42DocValuesFormat, TestLucene40DocValuesFormat, Nested1, Nested2, Nested3, TestComplexExplanationsOfNonMatches, TestBlockPostingsFormat2, TestIndexWriterMerging, Test2BPositions, TestSegmentReader, TestWildcard, TestConstantScoreQuery, TestSpanExplanationsOfNonMatches, TestFilterIterator, TestSloppyPhraseQuery, TestTermsEnum2, Nested1, TestTopScoreDocCollector, TestForceMergeForever, TestIndexWriter, TestScoreCachingWrappingScorer, Nested1, TestSetOnce, TestDemo, TestTermRangeQuery, TestDocValuesFormat, TestIndexWriterReader, TestBytesStore, TestIndexWriterExceptions, TestDeletionPolicy, TestBooleanOr, TestBasics, TestFlushByRamOrCountsPolicy, TestIndexWriterDelete, TestNRTThreads, TestStressAdvance, TestConcurrentMergeScheduler, TestHighCompressionMode, TestDocumentsWriterStallControl, TestTopDocsMerge, TestCompoundFile, TestLongPostings, TestMinimize, TestDocTermOrdsRewriteMethod]
> [junit4:junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
> [junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestDocTermOrdsRewriteMethod -Dtests.seed=2923E24EEDCF7E1F -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=tr -Dtests.timezone=Etc/GMT -Dtests.file.encoding=ISO8859-1
> [junit4:junit4] ERROR   0.00s J0 | TestDocTermOrdsRewriteMethod (suite) <<<
> [junit4:junit4]    > Throwable #1: java.lang.Exception: Suite timeout exceeded (>= 7200000 msec).
> [junit4:junit4]    >    at __randomizedtesting.SeedInfo.seed([2923E24EEDCF7E1F]:0)
> [junit4:junit4] Completed on J0 in 7223.18s, 2 tests, 2 errors <<< FAILURES!
>
> [...truncated 11 lines...]
> BUILD FAILED
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:388: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:361: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/build.xml:50: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1213: The following error occurred while executing this line:
> /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:877: There were test failures: 351 suites, 2183 tests, 1 suite-level error, 1 error, 29 ignored (17 assumptions)
>
> Total time: 144 minutes 26 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure
> Sending email for trigger: Failure
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 206 - Still Failing

Apache Jenkins Server-2
In reply to this post by Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/206/

3 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
        at __randomizedtesting.SeedInfo.seed([8E2BB79336CEE2DA:FCD398B419182E6]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
        at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
        at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)


FAILED:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:11599/collection1lastClient and got 5 from http://127.0.0.1:11603/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:11599/collection1lastClient and got 5 from http://127.0.0.1:11603/collection1
        at __randomizedtesting.SeedInfo.seed([8E2BB79336CEE2DA:FCD398B419182E6]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:235)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)


FAILED:  org.apache.solr.core.OpenCloseCoreStressTest.test10MinutesNew

Error Message:
Core 00006_core bad! expected:<490> but was:<476>

Stack Trace:
java.lang.AssertionError: Core 00006_core bad! expected:<490> but was:<476>
        at __randomizedtesting.SeedInfo.seed([8E2BB79336CEE2DA:B40E0FB24400086E]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.failNotEquals(Assert.java:647)
        at org.junit.Assert.assertEquals(Assert.java:128)
        at org.junit.Assert.assertEquals(Assert.java:472)
        at org.apache.solr.core.OpenCloseCoreStressTest.checkResults(OpenCloseCoreStressTest.java:300)
        at org.apache.solr.core.OpenCloseCoreStressTest.doStress(OpenCloseCoreStressTest.java:201)
        at org.apache.solr.core.OpenCloseCoreStressTest.test10MinutesNew(OpenCloseCoreStressTest.java:133)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8767 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T1579 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /k_s/
[junit4:junit4]   2> 5 T1579 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691
[junit4:junit4]   2> 6 T1579 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 6 T1580 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 9 T1580 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 9 T1580 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 9 T1580 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 10 T1580 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11 T1580 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 107 T1579 oasc.ZkTestServer.run start zk server on port:29533
[junit4:junit4]   2> 107 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@296edfc3
[junit4:junit4]   2> 108 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 108 T1585 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 109 T1585 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 109 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:25851
[junit4:junit4]   2> 109 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:25851
[junit4:junit4]   2> 110 T1583 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 112 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20000 with negotiated timeout 10000 for client /140.211.11.196:25851
[junit4:junit4]   2> 112 T1585 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20000, negotiated timeout = 10000
[junit4:junit4]   2> 112 T1586 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@296edfc3 name:ZooKeeperConnection Watcher:127.0.0.1:29533 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 112 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 113 T1579 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 115 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d20000
[junit4:junit4]   2> 129 T1586 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 129 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d20000 closed
[junit4:junit4]   2> 129 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:25851 which had sessionid 0x13d60aca3d20000
[junit4:junit4]   2> 130 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@5a0d308e
[junit4:junit4]   2> 131 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 131 T1587 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 131 T1587 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 132 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:36231
[junit4:junit4]   2> 132 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:36231
[junit4:junit4]   2> 133 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20001 with negotiated timeout 10000 for client /140.211.11.196:36231
[junit4:junit4]   2> 133 T1587 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20001, negotiated timeout = 10000
[junit4:junit4]   2> 133 T1588 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5a0d308e name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 133 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 134 T1579 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 143 T1579 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 146 T1579 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 148 T1579 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 156 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 156 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 161 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 162 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 266 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 267 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 269 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 270 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 280 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 281 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 283 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 284 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 287 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 288 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 291 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 291 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 294 T1579 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 295 T1579 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 297 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d20001
[junit4:junit4]   2> 305 T1588 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 305 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:36231 which had sessionid 0x13d60aca3d20001
[junit4:junit4]   2> 305 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d20001 closed
[junit4:junit4]   2> 486 T1579 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 489 T1579 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:60600
[junit4:junit4]   2> 490 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 490 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 491 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992
[junit4:junit4]   2> 491 T1579 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/solr.xml
[junit4:junit4]   2> 492 T1579 oasc.CoreContainer.<init> New CoreContainer 1553549657
[junit4:junit4]   2> 492 T1579 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/'
[junit4:junit4]   2> 493 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/'
[junit4:junit4]   2> 543 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 543 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 544 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 544 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 545 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 545 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 545 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 546 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 546 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 547 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 568 T1579 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 586 T1579 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:29533/solr
[junit4:junit4]   2> 587 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 587 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@53805ade
[junit4:junit4]   2> 588 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 588 T1598 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 589 T1598 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 590 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:45936
[junit4:junit4]   2> 590 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:45936
[junit4:junit4]   2> 592 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20002 with negotiated timeout 20000 for client /140.211.11.196:45936
[junit4:junit4]   2> 592 T1598 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20002, negotiated timeout = 20000
[junit4:junit4]   2> 592 T1599 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@53805ade name:ZooKeeperConnection Watcher:127.0.0.1:29533 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 592 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 593 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d20002
[junit4:junit4]   2> 604 T1599 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 604 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:45936 which had sessionid 0x13d60aca3d20002
[junit4:junit4]   2> 604 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d20002 closed
[junit4:junit4]   2> 605 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 617 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@39d16e44
[junit4:junit4]   2> 618 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 618 T1600 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 618 T1600 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 619 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:35901
[junit4:junit4]   2> 619 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:35901
[junit4:junit4]   2> 620 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20003 with negotiated timeout 20000 for client /140.211.11.196:35901
[junit4:junit4]   2> 620 T1600 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20003, negotiated timeout = 20000
[junit4:junit4]   2> 620 T1601 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39d16e44 name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 621 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 623 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 631 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 639 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60600_k_s
[junit4:junit4]   2> 640 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:60600_k_s Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:60600_k_s
[junit4:junit4]   2> 695 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60600_k_s
[junit4:junit4]   2> 698 T1579 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 735 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 738 T1579 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 807 T1579 oasc.Overseer.start Overseer (id=89333862216368131-127.0.0.1:60600_k_s-n_0000000000) starting
[junit4:junit4]   2> 808 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 811 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 869 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 870 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 880 T1603 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 882 T1579 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 885 T1579 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 915 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 917 T1602 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 921 T1604 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/collection1
[junit4:junit4]   2> 921 T1604 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 922 T1604 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 922 T1604 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 923 T1604 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/collection1/'
[junit4:junit4]   2> 924 T1604 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/collection1/lib/README' to classloader
[junit4:junit4]   2> 925 T1604 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 972 T1604 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 1036 T1604 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1137 T1604 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1143 T1604 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1742 T1604 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1753 T1604 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1757 T1604 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1770 T1604 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1774 T1604 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1779 T1604 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1779 T1604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1780 T1604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1780 T1604 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1781 T1604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1781 T1604 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1781 T1604 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1782 T1604 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363126559992/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/control/data/
[junit4:junit4]   2> 1782 T1604 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 1782 T1604 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1783 T1604 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/control/data forceNew: false
[junit4:junit4]   2> 1783 T1604 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/control/data/index/
[junit4:junit4]   2> 1783 T1604 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1784 T1604 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/control/data/index forceNew: false
[junit4:junit4]   2> 1787 T1604 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46a695aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@6227aca),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1787 T1604 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1789 T1604 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1790 T1604 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1790 T1604 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1791 T1604 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1791 T1604 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1791 T1604 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1792 T1604 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1792 T1604 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1793 T1604 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1799 T1604 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1805 T1604 oass.SolrIndexSearcher.<init> Opening Searcher@33c5191b main
[junit4:junit4]   2> 1805 T1604 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/control/data/tlog
[junit4:junit4]   2> 1806 T1604 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1806 T1604 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1810 T1605 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@33c5191b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1811 T1604 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1811 T1604 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1812 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2420 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 2421 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:60600_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:60600/k_s"}
[junit4:junit4]   2> 2421 T1602 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2421 T1602 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2422 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2439 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2851 T1604 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2851 T1604 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60600/k_s collection:control_collection shard:shard1
[junit4:junit4]   2> 2851 T1604 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2868 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2869 T1604 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2869 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2878 T1604 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2878 T1604 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2878 T1604 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60600/k_s/collection1/
[junit4:junit4]   2> 2878 T1604 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2879 T1604 oasc.SyncStrategy.syncToMe http://127.0.0.1:60600/k_s/collection1/ has no replicas
[junit4:junit4]   2> 2879 T1604 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60600/k_s/collection1/
[junit4:junit4]   2> 2879 T1604 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2884 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3951 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 3975 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 4016 T1604 oasc.ZkController.register We are http://127.0.0.1:60600/k_s/collection1/ and leader is http://127.0.0.1:60600/k_s/collection1/
[junit4:junit4]   2> 4016 T1604 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60600/k_s
[junit4:junit4]   2> 4016 T1604 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 4016 T1604 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 4017 T1604 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 4019 T1604 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 4020 T1579 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 4020 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4021 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4038 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 4039 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@45fd4f3a
[junit4:junit4]   2> 4040 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4040 T1607 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4041 T1607 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 4041 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:14278
[junit4:junit4]   2> 4041 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:14278
[junit4:junit4]   2> 4042 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20004 with negotiated timeout 10000 for client /140.211.11.196:14278
[junit4:junit4]   2> 4042 T1607 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20004, negotiated timeout = 10000
[junit4:junit4]   2> 4043 T1608 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45fd4f3a name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4043 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4044 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 4046 T1579 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4251 T1579 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4254 T1579 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:12427
[junit4:junit4]   2> 4255 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4255 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4256 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732
[junit4:junit4]   2> 4256 T1579 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/solr.xml
[junit4:junit4]   2> 4257 T1579 oasc.CoreContainer.<init> New CoreContainer 1641945772
[junit4:junit4]   2> 4257 T1579 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/'
[junit4:junit4]   2> 4258 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/'
[junit4:junit4]   2> 4322 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4323 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4323 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4324 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4324 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4325 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4325 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4325 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4326 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4326 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4350 T1579 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4371 T1579 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:29533/solr
[junit4:junit4]   2> 4371 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4372 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@cc9a3a7
[junit4:junit4]   2> 4373 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4373 T1618 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4374 T1618 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 4374 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:41269
[junit4:junit4]   2> 4375 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:41269
[junit4:junit4]   2> 4377 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20005 with negotiated timeout 20000 for client /140.211.11.196:41269
[junit4:junit4]   2> 4377 T1618 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20005, negotiated timeout = 20000
[junit4:junit4]   2> 4377 T1619 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cc9a3a7 name:ZooKeeperConnection Watcher:127.0.0.1:29533 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4377 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4378 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d20005
[junit4:junit4]   2> 4387 T1619 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4387 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:41269 which had sessionid 0x13d60aca3d20005
[junit4:junit4]   2> 4387 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d20005 closed
[junit4:junit4]   2> 4388 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4400 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@5ad6dc32
[junit4:junit4]   2> 4401 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4401 T1620 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4402 T1620 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 4402 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:29229
[junit4:junit4]   2> 4402 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:29229
[junit4:junit4]   2> 4404 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20006 with negotiated timeout 20000 for client /140.211.11.196:29229
[junit4:junit4]   2> 4404 T1620 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20006, negotiated timeout = 20000
[junit4:junit4]   2> 4404 T1621 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5ad6dc32 name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4404 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4405 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4406 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4414 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 5417 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:12427_k_s
[junit4:junit4]   2> 5418 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:12427_k_s Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:12427_k_s
[junit4:junit4]   2> 5424 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:12427_k_s
[junit4:junit4]   2> 5427 T1621 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5428 T1608 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5428 T1601 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5428 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5446 T1622 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/collection1
[junit4:junit4]   2> 5447 T1622 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5447 T1622 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5447 T1622 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5448 T1622 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/collection1/'
[junit4:junit4]   2> 5449 T1622 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/collection1/lib/README' to classloader
[junit4:junit4]   2> 5449 T1622 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5486 T1622 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 5488 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 5488 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:60600_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:60600/k_s"}
[junit4:junit4]   2> 5510 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5510 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5510 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5534 T1622 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5635 T1622 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5639 T1622 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6115 T1622 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6125 T1622 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6129 T1622 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6140 T1622 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6144 T1622 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6147 T1622 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6148 T1622 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6149 T1622 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6149 T1622 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6150 T1622 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6150 T1622 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6150 T1622 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6150 T1622 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty1/
[junit4:junit4]   2> 6151 T1622 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 6151 T1622 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6152 T1622 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty1 forceNew: false
[junit4:junit4]   2> 6152 T1622 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty1/index/
[junit4:junit4]   2> 6152 T1622 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6152 T1622 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty1/index forceNew: false
[junit4:junit4]   2> 6155 T1622 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@618e5b lockFactory=org.apache.lucene.store.NativeFSLockFactory@263676bd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6155 T1622 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6157 T1622 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6157 T1622 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6157 T1622 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6158 T1622 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6158 T1622 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6158 T1622 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6159 T1622 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6159 T1622 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6159 T1622 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6165 T1622 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6170 T1622 oass.SolrIndexSearcher.<init> Opening Searcher@7b2f86b2 main
[junit4:junit4]   2> 6170 T1622 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty1/tlog
[junit4:junit4]   2> 6171 T1622 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6171 T1622 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6174 T1623 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b2f86b2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6175 T1622 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6175 T1622 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7014 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 7015 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:12427_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:12427/k_s"}
[junit4:junit4]   2> 7015 T1602 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 7015 T1602 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 7054 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7054 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7054 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7178 T1622 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7178 T1622 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:12427/k_s collection:collection1 shard:shard1
[junit4:junit4]   2> 7179 T1622 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7195 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7196 T1622 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7197 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7207 T1622 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7207 T1622 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7207 T1622 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:12427/k_s/collection1/
[junit4:junit4]   2> 7207 T1622 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7207 T1622 oasc.SyncStrategy.syncToMe http://127.0.0.1:12427/k_s/collection1/ has no replicas
[junit4:junit4]   2> 7208 T1622 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:12427/k_s/collection1/
[junit4:junit4]   2> 7208 T1622 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7211 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8558 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8588 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8588 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8588 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8597 T1622 oasc.ZkController.register We are http://127.0.0.1:12427/k_s/collection1/ and leader is http://127.0.0.1:12427/k_s/collection1/
[junit4:junit4]   2> 8597 T1622 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:12427/k_s
[junit4:junit4]   2> 8597 T1622 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8597 T1622 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8597 T1622 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8599 T1622 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8599 T1579 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8600 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8601 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8826 T1579 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8829 T1579 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:31824
[junit4:junit4]   2> 8830 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8830 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8831 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303
[junit4:junit4]   2> 8831 T1579 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/solr.xml
[junit4:junit4]   2> 8831 T1579 oasc.CoreContainer.<init> New CoreContainer 1415147648
[junit4:junit4]   2> 8832 T1579 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/'
[junit4:junit4]   2> 8833 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/'
[junit4:junit4]   2> 8893 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8893 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8894 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8894 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8895 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8895 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8895 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8896 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8896 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8897 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8921 T1579 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8942 T1579 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:29533/solr
[junit4:junit4]   2> 8942 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8943 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@41c842ef
[junit4:junit4]   2> 8944 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8944 T1634 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8945 T1634 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 8946 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:60862
[junit4:junit4]   2> 8946 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:60862
[junit4:junit4]   2> 8948 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20007 with negotiated timeout 20000 for client /140.211.11.196:60862
[junit4:junit4]   2> 8948 T1634 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20007, negotiated timeout = 20000
[junit4:junit4]   2> 8948 T1635 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41c842ef name:ZooKeeperConnection Watcher:127.0.0.1:29533 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8949 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8950 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d20007
[junit4:junit4]   2> 8951 T1635 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8951 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:60862 which had sessionid 0x13d60aca3d20007
[junit4:junit4]   2> 8951 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d20007 closed
[junit4:junit4]   2> 8952 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8964 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@bb6152b
[junit4:junit4]   2> 8965 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8965 T1636 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8966 T1636 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 8966 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:18888
[junit4:junit4]   2> 8966 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:18888
[junit4:junit4]   2> 8968 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20008 with negotiated timeout 20000 for client /140.211.11.196:18888
[junit4:junit4]   2> 8968 T1636 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20008, negotiated timeout = 20000
[junit4:junit4]   2> 8968 T1637 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bb6152b name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8968 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8969 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8970 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8979 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 9982 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:31824_k_s
[junit4:junit4]   2> 9983 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:31824_k_s Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:31824_k_s
[junit4:junit4]   2> 9991 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:31824_k_s
[junit4:junit4]   2> 10015 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 10015 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 10016 T1601 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10016 T1637 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10016 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10017 T1621 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10017 T1608 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10022 T1638 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/collection1
[junit4:junit4]   2> 10022 T1638 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 10022 T1638 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 10022 T1638 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 10023 T1638 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/collection1/'
[junit4:junit4]   2> 10024 T1638 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/collection1/lib/README' to classloader
[junit4:junit4]   2> 10025 T1638 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 10064 T1638 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 10099 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 10100 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:12427_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:12427/k_s"}
[junit4:junit4]   2> 10107 T1638 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10160 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10160 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10160 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10237 T1638 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10237 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10243 T1638 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 11071 T1638 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 11082 T1638 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 11087 T1638 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 11102 T1638 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 11107 T1638 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 11111 T1638 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 11112 T1638 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 11113 T1638 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 11114 T1638 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 11115 T1638 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 11115 T1638 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 11115 T1638 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 11116 T1638 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363126568303/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty2/
[junit4:junit4]   2> 11116 T1638 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 11116 T1638 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 11117 T1638 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty2 forceNew: false
[junit4:junit4]   2> 11117 T1638 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty2/index/
[junit4:junit4]   2> 11117 T1638 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 11118 T1638 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty2/index forceNew: false
[junit4:junit4]   2> 11121 T1638 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e0e20ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@126fcb77),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11121 T1638 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 11124 T1638 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 11124 T1638 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 11125 T1638 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 11125 T1638 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 11126 T1638 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 11126 T1638 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 11126 T1638 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 11127 T1638 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 11127 T1638 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 11135 T1638 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 11142 T1638 oass.SolrIndexSearcher.<init> Opening Searcher@24972429 main
[junit4:junit4]   2> 11143 T1638 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty2/tlog
[junit4:junit4]   2> 11144 T1638 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 11144 T1638 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 11148 T1639 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24972429 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 11150 T1638 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 11150 T1638 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11688 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 11688 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:31824_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:31824/k_s"}
[junit4:junit4]   2> 11688 T1602 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 11689 T1602 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 11710 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11710 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11710 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11710 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12157 T1638 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 12157 T1638 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:31824/k_s collection:collection1 shard:shard2
[junit4:junit4]   2> 12158 T1638 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 12176 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 12182 T1638 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 12183 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12194 T1638 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 12195 T1638 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 12195 T1638 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:31824/k_s/collection1/
[junit4:junit4]   2> 12195 T1638 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 12196 T1638 oasc.SyncStrategy.syncToMe http://127.0.0.1:31824/k_s/collection1/ has no replicas
[junit4:junit4]   2> 12196 T1638 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:31824/k_s/collection1/
[junit4:junit4]   2> 12196 T1638 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 12201 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13217 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 13247 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13247 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13247 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13247 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13290 T1638 oasc.ZkController.register We are http://127.0.0.1:31824/k_s/collection1/ and leader is http://127.0.0.1:31824/k_s/collection1/
[junit4:junit4]   2> 13290 T1638 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:31824/k_s
[junit4:junit4]   2> 13290 T1638 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13290 T1638 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 13290 T1638 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13293 T1638 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 13294 T1579 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 13294 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13295 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13596 T1579 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 13598 T1579 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:22661
[junit4:junit4]   2> 13600 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13600 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13601 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995
[junit4:junit4]   2> 13601 T1579 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/solr.xml
[junit4:junit4]   2> 13601 T1579 oasc.CoreContainer.<init> New CoreContainer 748788295
[junit4:junit4]   2> 13602 T1579 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/'
[junit4:junit4]   2> 13603 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/'
[junit4:junit4]   2> 13657 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 13658 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13659 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 13659 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13659 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13660 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13660 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13661 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13661 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13662 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13703 T1579 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13731 T1579 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:29533/solr
[junit4:junit4]   2> 13732 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13733 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5d233a8e
[junit4:junit4]   2> 13734 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13734 T1650 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13735 T1650 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 13735 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32282
[junit4:junit4]   2> 13736 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32282
[junit4:junit4]   2> 13739 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d20009 with negotiated timeout 20000 for client /140.211.11.196:32282
[junit4:junit4]   2> 13739 T1650 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d20009, negotiated timeout = 20000
[junit4:junit4]   2> 13739 T1651 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d233a8e name:ZooKeeperConnection Watcher:127.0.0.1:29533 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13739 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13741 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d20009
[junit4:junit4]   2> 13742 T1651 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13742 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:32282 which had sessionid 0x13d60aca3d20009
[junit4:junit4]   2> 13742 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d20009 closed
[junit4:junit4]   2> 13743 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13757 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@119b01d1
[junit4:junit4]   2> 13758 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13759 T1652 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13759 T1652 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 13759 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:50964
[junit4:junit4]   2> 13760 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:50964
[junit4:junit4]   2> 13761 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d2000a with negotiated timeout 20000 for client /140.211.11.196:50964
[junit4:junit4]   2> 13761 T1652 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d2000a, negotiated timeout = 20000
[junit4:junit4]   2> 13761 T1653 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@119b01d1 name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13761 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13762 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d2000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13764 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d2000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13773 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 14754 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 14755 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:31824_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:31824/k_s"}
[junit4:junit4]   2> 14782 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14782 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14782 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14782 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14782 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14782 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:22661_k_s
[junit4:junit4]   2> 14784 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d2000a type:delete cxid:0xd zxid:0x96 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:22661_k_s Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:22661_k_s
[junit4:junit4]   2> 14797 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:22661_k_s
[junit4:junit4]   2> 14805 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14805 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14805 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14805 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14805 T1601 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14806 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14807 T1608 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14807 T1621 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14807 T1637 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14807 T1653 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14822 T1654 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/collection1
[junit4:junit4]   2> 14823 T1654 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14823 T1654 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14824 T1654 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14825 T1654 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/collection1/'
[junit4:junit4]   2> 14827 T1654 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/collection1/lib/README' to classloader
[junit4:junit4]   2> 14828 T1654 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14925 T1654 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 15018 T1654 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 15119 T1654 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 15130 T1654 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15945 T1654 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15960 T1654 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15966 T1654 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15988 T1654 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15995 T1654 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 16004 T1654 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 16006 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 16007 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 16008 T1654 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 16009 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 16009 T1654 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 16009 T1654 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 16009 T1654 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363126572995/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3/
[junit4:junit4]   2> 16010 T1654 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 16010 T1654 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 16011 T1654 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3 forceNew: false
[junit4:junit4]   2> 16011 T1654 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3/index/
[junit4:junit4]   2> 16011 T1654 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 16012 T1654 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3/index forceNew: false
[junit4:junit4]   2> 16015 T1654 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3663c64f lockFactory=org.apache.lucene.store.NativeFSLockFactory@c0d1704),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 16015 T1654 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 16018 T1654 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 16018 T1654 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 16019 T1654 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 16019 T1654 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 16020 T1654 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 16020 T1654 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 16020 T1654 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 16021 T1654 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 16021 T1654 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 16032 T1654 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 16042 T1654 oass.SolrIndexSearcher.<init> Opening Searcher@313756e3 main
[junit4:junit4]   2> 16042 T1654 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3/tlog
[junit4:junit4]   2> 16043 T1654 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 16043 T1654 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 16047 T1655 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@313756e3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 16049 T1654 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 16049 T1654 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16300 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 16300 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:22661_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:22661/k_s"}
[junit4:junit4]   2> 16301 T1602 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 16301 T1602 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16315 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16315 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16316 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16315 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16316 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17052 T1654 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 17052 T1654 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:22661/k_s collection:collection1 shard:shard1
[junit4:junit4]   2> 17067 T1654 oasc.ZkController.register We are http://127.0.0.1:22661/k_s/collection1/ and leader is http://127.0.0.1:12427/k_s/collection1/
[junit4:junit4]   2> 17067 T1654 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:22661/k_s
[junit4:junit4]   2> 17067 T1654 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 17067 T1654 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C111 name=collection1 org.apache.solr.core.SolrCore@41887b1b url=http://127.0.0.1:22661/k_s/collection1 node=127.0.0.1:22661_k_s C111_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:22661_k_s, base_url=http://127.0.0.1:22661/k_s}
[junit4:junit4]   2> 17068 T1656 C111 P22661 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 17069 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 17069 T1654 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 17069 T1656 C111 P22661 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 17070 T1656 C111 P22661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 17070 T1579 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 17071 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 17072 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 17074 T1656 C111 P22661 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 17320 T1579 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 17323 T1579 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:46270
[junit4:junit4]   2> 17324 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 17324 T1579 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 17325 T1579 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775
[junit4:junit4]   2> 17325 T1579 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/solr.xml
[junit4:junit4]   2> 17326 T1579 oasc.CoreContainer.<init> New CoreContainer 885000855
[junit4:junit4]   2> 17326 T1579 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/'
[junit4:junit4]   2> 17327 T1579 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/'
[junit4:junit4]   2> 17377 T1579 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 17378 T1579 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 17379 T1579 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 17379 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 17379 T1579 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 17380 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 17380 T1579 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 17381 T1579 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 17381 T1579 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 17382 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 17404 T1579 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 17422 T1579 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:29533/solr
[junit4:junit4]   2> 17423 T1579 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 17423 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@7c5a78d5
[junit4:junit4]   2> 17424 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 17424 T1667 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 17425 T1667 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 17426 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:53767
[junit4:junit4]   2> 17426 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:53767
[junit4:junit4]   2> 17428 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d2000b with negotiated timeout 20000 for client /140.211.11.196:53767
[junit4:junit4]   2> 17428 T1667 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d2000b, negotiated timeout = 20000
[junit4:junit4]   2> 17428 T1668 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c5a78d5 name:ZooKeeperConnection Watcher:127.0.0.1:29533 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 17428 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 17429 T1584 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d60aca3d2000b
[junit4:junit4]   2> 17430 T1668 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 17430 T1581 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:53767 which had sessionid 0x13d60aca3d2000b
[junit4:junit4]   2> 17430 T1579 oaz.ZooKeeper.close Session: 0x13d60aca3d2000b closed
[junit4:junit4]   2> 17431 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 17442 T1579 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:29533/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@30c92080
[junit4:junit4]   2> 17443 T1579 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 17444 T1669 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:29533. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 17444 T1669 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:29533, initiating session
[junit4:junit4]   2> 17444 T1581 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:21445
[junit4:junit4]   2> 17445 T1581 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:21445
[junit4:junit4]   2> 17446 T1583 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d60aca3d2000c with negotiated timeout 20000 for client /140.211.11.196:21445
[junit4:junit4]   2> 17446 T1669 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:29533, sessionid = 0x13d60aca3d2000c, negotiated timeout = 20000
[junit4:junit4]   2> 17446 T1670 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30c92080 name:ZooKeeperConnection Watcher:127.0.0.1:29533/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 17446 T1579 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 17447 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d2000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 17448 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d2000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 17458 T1579 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 17821 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 17822 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:22661_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:22661/k_s"}
[junit4:junit4]   2> 17837 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17837 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17837 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17837 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17837 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17837 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18092 T1611 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:22661_k_s_collection1&state=recovering&nodeName=127.0.0.1:22661_k_s&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 18461 T1579 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:46270_k_s
[junit4:junit4]   2> 18462 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d2000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:46270_k_s Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:46270_k_s
[junit4:junit4]   2> 18463 T1579 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:46270_k_s
[junit4:junit4]   2> 18466 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18466 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18466 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18466 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18466 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 18466 T1601 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18467 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18468 T1653 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18468 T1670 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18468 T1608 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18468 T1637 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18469 T1621 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 18482 T1671 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/collection1
[junit4:junit4]   2> 18482 T1671 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 18483 T1671 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 18483 T1671 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 18484 T1671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/collection1/'
[junit4:junit4]   2> 18485 T1671 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 18486 T1671 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/collection1/lib/README' to classloader
[junit4:junit4]   2> 18534 T1671 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 18592 T1671 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 18692 T1671 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 18698 T1671 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 19184 T1671 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 19191 T1671 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 19194 T1671 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 19205 T1671 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19209 T1671 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19212 T1671 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19213 T1671 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19213 T1671 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19213 T1671 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19214 T1671 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19214 T1671 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19214 T1671 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19214 T1671 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363126576775/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4/
[junit4:junit4]   2> 19215 T1671 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 19215 T1671 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19215 T1671 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4 forceNew: false
[junit4:junit4]   2> 19216 T1671 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4/index/
[junit4:junit4]   2> 19216 T1671 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 19216 T1671 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4/index forceNew: false
[junit4:junit4]   2> 19218 T1671 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f094ba2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ae9e7a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 19218 T1671 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 19220 T1671 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 19220 T1671 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 19220 T1671 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 19221 T1671 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 19221 T1671 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 19221 T1671 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 19221 T1671 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 19222 T1671 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 19222 T1671 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 19226 T1671 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 19230 T1671 oass.SolrIndexSearcher.<init> Opening Searcher@2b027c27 main
[junit4:junit4]   2> 19230 T1671 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4/tlog
[junit4:junit4]   2> 19231 T1671 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 19231 T1671 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 19234 T1672 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2b027c27 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 19235 T1671 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 19235 T1671 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19343 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 19344 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:46270_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:46270/k_s"}
[junit4:junit4]   2> 19344 T1602 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 19344 T1602 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 19356 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19356 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19356 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19356 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19356 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19356 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2>  C111_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:22661_k_s, base_url=http://127.0.0.1:22661/k_s}
[junit4:junit4]   2> 20094 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:12427/k_s/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 20094 T1656 C111 P22661 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:22661/k_s START replicas=[http://127.0.0.1:12427/k_s/collection1/] nUpdates=100
[junit4:junit4]   2> 20094 T1656 C111 P22661 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 20095 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 20096 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 20096 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 20096 T1656 C111 P22661 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 20096 T1656 C111 P22661 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:12427/k_s/collection1/. core=collection1
[junit4:junit4]   2> 20096 T1656 C111 P22661 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C112 name=collection1 org.apache.solr.core.SolrCore@e6ce4ef url=http://127.0.0.1:12427/k_s/collection1 node=127.0.0.1:12427_k_s C112_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:12427_k_s, base_url=http://127.0.0.1:12427/k_s, leader=true}
[junit4:junit4]   2> 20102 T1612 C112 P12427 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 20111 T1613 C112 P12427 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 20113 T1613 C112 P12427 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@618e5b lockFactory=org.apache.lucene.store.NativeFSLockFactory@263676bd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20114 T1613 C112 P12427 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20114 T1613 C112 P12427 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@618e5b lockFactory=org.apache.lucene.store.NativeFSLockFactory@263676bd),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@618e5b lockFactory=org.apache.lucene.store.NativeFSLockFactory@263676bd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20115 T1613 C112 P12427 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20115 T1613 C112 P12427 oass.SolrIndexSearcher.<init> Opening Searcher@84765c7 realtime
[junit4:junit4]   2> 20116 T1613 C112 P12427 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 20116 T1613 C112 P12427 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 20117 T1656 C111 P22661 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20117 T1656 C111 P22661 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 20118 T1614 C112 P12427 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20119 T1614 C112 P12427 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4]   2> 20119 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 20119 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 20120 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 20122 T1614 C112 P12427 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4]   2> 20123 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 20124 T1656 C111 P22661 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty3/index.20130312191619809 forceNew: false
[junit4:junit4]   2> 20124 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@6ced093a lockFactory=org.apache.lucene.store.NativeFSLockFactory@240cc335) fullCopy=false
[junit4:junit4]   2> 20126 T1614 C112 P12427 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4]   2> 20127 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 20128 T1656 C111 P22661 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 20128 T1656 C111 P22661 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 20129 T1656 C111 P22661 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3663c64f lockFactory=org.apache.lucene.store.NativeFSLockFactory@c0d1704),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3663c64f lockFactory=org.apache.lucene.store.NativeFSLockFactory@c0d1704),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20130 T1656 C111 P22661 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20130 T1656 C111 P22661 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 20130 T1656 C111 P22661 oass.SolrIndexSearcher.<init> Opening Searcher@7804c48c main
[junit4:junit4]   2> 20131 T1655 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7804c48c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 20132 T1656 C111 P22661 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@6ced093a lockFactory=org.apache.lucene.store.NativeFSLockFactory@240cc335)
[junit4:junit4]   2> 20132 T1656 C111 P22661 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 20132 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 20132 T1656 C111 P22661 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 20132 T1656 C111 P22661 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20164 T1656 C111 P22661 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20238 T1671 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 20238 T1671 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:46270/k_s collection:collection1 shard:shard2
[junit4:junit4]   2> 20241 T1671 oasc.ZkController.register We are http://127.0.0.1:46270/k_s/collection1/ and leader is http://127.0.0.1:31824/k_s/collection1/
[junit4:junit4]   2> 20241 T1671 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:46270/k_s
[junit4:junit4]   2> 20241 T1671 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 20241 T1671 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C113 name=collection1 org.apache.solr.core.SolrCore@71692933 url=http://127.0.0.1:46270/k_s/collection1 node=127.0.0.1:46270_k_s C113_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:46270_k_s, base_url=http://127.0.0.1:46270/k_s}
[junit4:junit4]   2> 20242 T1675 C113 P46270 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 20242 T1671 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 20243 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 20243 T1675 C113 P46270 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 20243 T1675 C113 P46270 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20243 T1579 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 20244 T1579 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 20244 T1675 C113 P46270 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20244 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20258 T1579 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 20259 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20269 T1615 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/onenodecollectioncore
[junit4:junit4]   2> 20269 T1615 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4]   2> 20270 T1615 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4]   2> 20270 T1615 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 20271 T1615 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 20272 T1615 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4]   2> 20274 T1615 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4]   2> 20276 T1615 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/onenodecollectioncore/'
[junit4:junit4]   2> 20337 T1615 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 20393 T1615 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 20395 T1615 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 20400 T1615 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 20861 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 20861 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:22661_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:22661/k_s"}
[junit4:junit4]   2> 20865 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:46270_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:46270/k_s"}
[junit4:junit4]   2> 20883 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20883 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20883 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20883 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20883 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20883 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20971 T1615 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 20979 T1615 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 20981 T1615 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 20993 T1615 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20997 T1615 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 21000 T1615 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 21001 T1615 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 21001 T1615 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 21002 T1615 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 21003 T1615 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 21003 T1615 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 21003 T1615 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 21003 T1615 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363126563732/onenodecollectioncore/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection/
[junit4:junit4]   2> 21004 T1615 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@6477eb97
[junit4:junit4]   2> 21004 T1615 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 21005 T1615 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection forceNew: false
[junit4:junit4]   2> 21005 T1615 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection/index/
[junit4:junit4]   2> 21005 T1615 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 21005 T1615 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection/index forceNew: false
[junit4:junit4]   2> 21007 T1615 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4647bf80 lockFactory=org.apache.lucene.store.NativeFSLockFactory@92cdfc1),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 21008 T1615 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 21009 T1615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 21010 T1615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 21010 T1615 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 21011 T1615 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 21011 T1615 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 21011 T1615 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 21011 T1615 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 21012 T1615 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 21012 T1615 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 21017 T1615 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 21021 T1615 oass.SolrIndexSearcher.<init> Opening Searcher@540db0f main
[junit4:junit4]   2> 21021 T1615 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection/tlog
[junit4:junit4]   2> 21022 T1615 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 21022 T1615 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 21024 T1677 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@540db0f main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 21025 T1615 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4]   2> 21259 T1627 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=127.0.0.1:46270_k_s_collection1&state=recovering&nodeName=127.0.0.1:46270_k_s&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 22389 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 22390 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":"none",
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"onenodecollectioncore",
[junit4:junit4]   2>  "collection":"onenodecollection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:12427_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:12427/k_s"}
[junit4:junit4]   2> 22390 T1602 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4]   2> 22391 T1602 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 22403 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22404 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22404 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22403 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22404 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22404 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23029 T1615 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4]   2> 23030 T1615 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:12427/k_s collection:onenodecollection shard:shard1
[junit4:junit4]   2> 23031 T1615 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4]   2> 23047 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:delete cxid:0xaa zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4]   2> 23048 T1615 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 23048 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:create cxid:0xab zxid:0xc6 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 23058 T1615 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 23059 T1615 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 23059 T1615 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:12427/k_s/onenodecollectioncore/
[junit4:junit4]   2> 23059 T1615 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 23060 T1615 oasc.SyncStrategy.syncToMe http://127.0.0.1:12427/k_s/onenodecollectioncore/ has no replicas
[junit4:junit4]   2> 23060 T1615 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:12427/k_s/onenodecollectioncore/
[junit4:junit4]   2> 23060 T1615 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4]   2> 23064 T1584 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d60aca3d20006 type:create cxid:0xb5 zxid:0xca txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2>  C113_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:46270_k_s, base_url=http://127.0.0.1:46270/k_s}
[junit4:junit4]   2> 23260 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:31824/k_s/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 23260 T1675 C113 P46270 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:46270/k_s START replicas=[http://127.0.0.1:31824/k_s/collection1/] nUpdates=100
[junit4:junit4]   2> 23260 T1675 C113 P46270 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 23261 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 23261 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 23261 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 23261 T1675 C113 P46270 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 23261 T1675 C113 P46270 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:31824/k_s/collection1/. core=collection1
[junit4:junit4]   2> 23262 T1675 C113 P46270 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C114 name=collection1 org.apache.solr.core.SolrCore@66176cd url=http://127.0.0.1:31824/k_s/collection1 node=127.0.0.1:31824_k_s C114_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:31824_k_s, base_url=http://127.0.0.1:31824/k_s, leader=true}
[junit4:junit4]   2> 23268 T1628 C114 P31824 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 23277 T1629 C114 P31824 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23280 T1629 C114 P31824 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e0e20ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@126fcb77),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 23280 T1629 C114 P31824 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 23281 T1629 C114 P31824 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e0e20ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@126fcb77),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e0e20ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@126fcb77),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23281 T1629 C114 P31824 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23282 T1629 C114 P31824 oass.SolrIndexSearcher.<init> Opening Searcher@7628ce7f realtime
[junit4:junit4]   2> 23282 T1629 C114 P31824 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23282 T1629 C114 P31824 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 23283 T1675 C113 P46270 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23283 T1675 C113 P46270 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 23285 T1630 C114 P31824 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23285 T1630 C114 P31824 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4]   2> 23286 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 23286 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 23286 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 23288 T1630 C114 P31824 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4]   2> 23289 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 23290 T1675 C113 P46270 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363126559691/jetty4/index.20130312191622975 forceNew: false
[junit4:junit4]   2> 23290 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@6492ba43 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d9131fa) fullCopy=false
[junit4:junit4]   2> 23293 T1630 C114 P31824 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4]   2> 23294 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 23295 T1675 C113 P46270 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 23295 T1675 C113 P46270 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 23297 T1675 C113 P46270 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f094ba2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ae9e7a2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f094ba2 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ae9e7a2),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23298 T1675 C113 P46270 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23298 T1675 C113 P46270 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 23298 T1675 C113 P46270 oass.SolrIndexSearcher.<init> Opening Searcher@3df8d83 main
[junit4:junit4]   2> 23299 T1672 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3df8d83 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 23300 T1675 C113 P46270 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@6492ba43 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5d9131fa)
[junit4:junit4]   2> 23300 T1675 C113 P46270 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 23300 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 23300 T1675 C113 P46270 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 23300 T1675 C113 P46270 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23302 T1675 C113 P46270 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 23908 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23928 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:46270_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:46270/k_s"}
[junit4:junit4]   2> 23932 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23932 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23932 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23932 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23932 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23932 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23984 T1615 oasc.ZkController.register We are http://127.0.0.1:12427/k_s/onenodecollectioncore/ and leader is http://127.0.0.1:12427/k_s/onenodecollectioncore/
[junit4:junit4]   2> 23984 T1615 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:12427/k_s
[junit4:junit4]   2> 23984 T1615 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 23985 T1615 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4]   2> 23987 T1615 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23987 T1615 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&roles=none&name=onenodecollectioncore&action=CREATE&collection=onenodecollection&dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363126559691/onenodecollection&wt=javabin&version=2} status=0 QTime=3718
[junit4:junit4]   2> 23988 T1579 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 23989 T1579 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 24991 T1579 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 25439 T1602 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 25440 T1602 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":"none",
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"onenodecollectioncore",
[junit4:junit4]   2>  "collection":"onenodecollection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:12427_k_s",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:12427/k_s"}
[junit4:junit4]   2> 25455 T1601 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25455 T1637 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25455 T1653 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25455 T1621 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25455 T1608 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25455 T1670 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 25993 T1579 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 25994 T1579 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4]   2> 25995 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C115 name=onenodecollectioncore org.apache.solr.core.SolrCore@2d8ec33 url=http://127.0.0.1:12427/k_s/onenodecollectioncore node=127.0.0.1:12427_k_s C115_STATE=coll:onenodecollection core:onenodecollectioncore props:{shard=shard1, roles=none, state=active, core=onenodecollectioncore, collection=onenodecollection, node_name=127.0.0.1:12427_k_s, base_url=http://127.0.0.1:12427/k_s, leader=true}
[junit4:junit4]   2> 26018 T1616 C115 P12427 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4647bf80 lockFactory=org.apache.lucene.store.NativeFSLockFactory@92cdfc1),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 26019 T1616 C115 P12427 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 26021 T1616 C115 P12427 /update {wt=javabin&version=2} {add=[1 (1429341822729060352)]} 0 5
[junit4:junit4]   2> 26028 T1616 C115 P12427 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26030 T1616 C115 P12427 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4647bf80 lockFactory=org.apache.lucene.store.NativeFSLockFactory@92cdfc1),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4647bf80 lockFactory=org.apache.lucene.store.NativeFSLockFactory@92cdfc1),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 26031 T1616 C115 P12427 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 26032 T1616 C115 P12427 oass.SolrIndexSearcher.<init> Opening Searcher@2b827602 main
[junit4:junit4]   2> 26033 T1616 C115 P12427 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26033 T1677 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@2b827602 main{StandardDirectoryReader(segments_2:3 _0(4.3):C1)}
[junit4:junit4]   2> 26034 T1616 C115 P12427 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 26038 T1616 C115 P12427 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4:junit4]   2> 26039 T1579 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 26054 T1616 C115 P12427 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=1
[junit4:junit4]   2> ASYNC  NEW_CORE C116 name=collection1 org.apache.solr.core.SolrCore@3893140d url=http://127.0.0.1:60600/k_s/collection1 node=127.0.0.1:60600_k_s C116_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:60600_k_s, base_url=http://127.0.0.1:60600/k_s, leader=true}
[junit4:junit4]   2> 26063 T1591 C116 P60600 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46a695aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@6227aca),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 26065 T1591 C116 P60600 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 26068 T1591 C116 P60600 /update {wt=javabin&version=2} {add=[1 (1429341822777294848)]} 0 6
[junit4:junit4]   2>  C111_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:22661_k_s, base_url=http://127.0.0.1:22661/k_s}
[junit4:junit4]   2> 26095 T1643 C111 P22661 /update {distrib.from=http://127.0.0.1:12427/k_s/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429341822797217792)]} 0 3
[junit4:junit4]   2> 26096 T1617 C112 P12427 /update {distrib.from=http://127.0.0.1:31824/k_s/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1429341822797217792)]} 0 15
[junit4:junit4]   2> 26096 T1633 C114 P31824 /update {wt=javabin&version=2} {add=[1]} 0 18
[junit4:junit4]   2> 26099 T1591 C116 P60600 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26104 T1591 C116 P60600 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46a695aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@6227aca),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@46a695aa lockFactory=org.apache.lucene.store.NativeFSLockFactory@6227aca),segFN=segments_2,generation=2,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 26104 T1591 C116 P60600 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, segments_2, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt]
[junit4:junit4]   2> 26106 T1591 C116 P60600 oass.SolrIndexSearcher.<init> Opening Searcher@145685b5 main
[junit4:junit4]   2> 26107 T1591 C116 P60600 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26107 T1605 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@145685b5 main{StandardDirectoryReader(segments_2:3 _0(4.3):C1)}
[junit4:junit4]   2> 26108 T1591 C116 P60600 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 26110 T1611 C112 P12427 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26114 T1611 C112 P12427 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@618e5b lockFactory=org.apache.lucene.store.NativeFSLockFactory@263676bd),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@618e5b lockFactory=org.apache.lucene.store.NativeFSLockFactory@263676bd),segFN=segments_3,generation=3,filenames=[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt, segments_3]
[junit4:junit4]   2> 26115 T1611 C112 P12427 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.fnm, _0.frq, _0.tim, _0_nrm.cfe, _0.fdx, _0_nrm.cfs, _0.si, _0.prx, _0.tip, _0.fdt, segments_3]
[junit4:junit4]   2> 26116 T1611 C112 P12427 oass.SolrIndexSearcher.<init> Opening Searcher@23cd9ff1 main
[junit4:junit4]   2> 26117 T1611 C112 P12427 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26117 T1623 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@23cd9ff1 main{StandardDirectoryReader(segments_3:3:nrt _0(4.3):C1)}
[junit4:junit4]   2> 26118 T1611 C112 P12427 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:22661/k_s/collection1/, StdNode: http://127.0.0.1:31824/k_s/collection1/, StdNode: http://127.0.0.1:46270/k_s/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 26120 T1627 C114 P31824 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26120 T1643 C111 P22661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 26121 T1627 C114 P31824 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e0e20ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@126fcb77),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1e0e20ac lockFactory=org.apache.lucene.store.NativeFSLockFactory@126fcb77),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 26122 T1627 C114 P31824 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 26122 T1627 C114 P31824 oass.SolrIndexSearcher.<init> Opening Searcher@35bf7c57 main
[junit4:junit4]   2> 26122 T1627 C114 P31824 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 26123 T1639 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@35bf7c57 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 26124 T1627 C114 P31824 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 26126 T1643 C111 P22661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3663c64f lockFactory=org.apache.lucene.store.NativeFSLockFactory@c0d1704),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3663c64f lockFactory=org.apache.

[...truncated too long message...]

asc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00016_core/data/index
[junit4:junit4]   2> 824852 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00016_core/data/index
[junit4:junit4]   2> 824867 T2013 oasc.SolrCore.close [00017_core]  CLOSING SolrCore org.apache.solr.core.SolrCore@7c6517f6
[junit4:junit4]   2> 824871 T2013 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 824871 T2013 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 824872 T2013 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 824872 T2013 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 824874 T2013 oasc.SolrCore.closeSearcher [00017_core] Closing main searcher on request.
[junit4:junit4]   2> 824874 T2013 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00017_core/data
[junit4:junit4]   2> 824875 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00017_core/data
[junit4:junit4]   2> 824876 T2013 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00017_core/data/index
[junit4:junit4]   2> 824877 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00017_core/data/index
[junit4:junit4]   2> 824902 T2013 oasc.SolrCore.close [00018_core]  CLOSING SolrCore org.apache.solr.core.SolrCore@591798a5
[junit4:junit4]   2> 824906 T2013 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 824906 T2013 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 824907 T2013 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 824907 T2013 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 824909 T2013 oasc.SolrCore.closeSearcher [00018_core] Closing main searcher on request.
[junit4:junit4]   2> 824909 T2013 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00018_core/data
[junit4:junit4]   2> 824910 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00018_core/data
[junit4:junit4]   2> 824911 T2013 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00018_core/data/index
[junit4:junit4]   2> 824911 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00018_core/data/index
[junit4:junit4]   2> 824930 T2013 oasc.SolrCore.close [00019_core]  CLOSING SolrCore org.apache.solr.core.SolrCore@4abe206f
[junit4:junit4]   2> 824934 T2013 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=1,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
[junit4:junit4]   2> 824934 T2013 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 824935 T2013 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 824935 T2013 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 824937 T2013 oasc.SolrCore.closeSearcher [00019_core] Closing main searcher on request.
[junit4:junit4]   2> 824938 T2013 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data/index
[junit4:junit4]   2> 824939 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data/index
[junit4:junit4]   2> 824975 T2013 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data
[junit4:junit4]   2> 824975 T2013 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data
[junit4:junit4]   2> 824976 T2013 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 825513 T2013 oas.SolrTestCaseJ4.tearDown ###Ending test30SecondsNew
[junit4:junit4] IGNOR/A 0.00s J1 | OpenCloseCoreStressTest.test1HourNew
[junit4:junit4]    > Assumption #1: 'weekly' test group is disabled (@Weekly)
[junit4:junit4]   2> 825518 T2013 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {id=PostingsFormat(name=SimpleText), text=PostingsFormat(name=Memory doPackFST= false)}, docValues:{}, sim=DefaultSimilarity, locale=es_NI, timezone=Australia/Tasmania
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=208800320,total=530186240
[junit4:junit4]   2> NOTE: All tests run in this JVM: [ConvertedLegacyTest, TestPartialUpdateDeduplication, TestCSVLoader, TestJmxMonitoredMap, TestHashPartitioner, BadIndexSchemaTest, TestRTGBase, FileUtilsTest, SolrCoreCheckLockOnStartupTest, UniqFieldsUpdateProcessorFactoryTest, TestDFRSimilarityFactory, IndexSchemaRuntimeFieldTest, TestSystemIdResolver, TestComponentsName, TestElisionMultitermQuery, TestSolrJ, TestCollationKeyRangeQueries, TestFiltering, CollectionsAPIDistributedZkTest, XsltUpdateRequestHandlerTest, SuggesterWFSTTest, TestDefaultSimilarityFactory, SuggesterTSTTest, TestPropInject, HardAutoCommitTest, DocValuesMultiTest, DateMathParserTest, DirectSolrSpellCheckerTest, ZkSolrClientTest, LeaderElectionIntegrationTest, TestRecovery, URLClassifyProcessorTest, BasicZkTest, DefaultValueUpdateProcessorTest, TestJmxIntegration, TestAnalyzedSuggestions, ShardRoutingCustomTest, CurrencyFieldXmlFileTest, TestStressReorder, RequiredFieldsTest, SortByFunctionTest, NotRequiredUniqueKeyTest, DOMUtilTest, DocumentAnalysisRequestHandlerTest, TestLuceneMatchVersion, TestBinaryField, HighlighterConfigTest, HighlighterTest, NoCacheHeaderTest, TestMergePolicyConfig, FieldAnalysisRequestHandlerTest, CoreContainerCoreInitFailuresTest, OpenExchangeRatesOrgProviderTest, SolrPluginUtilsTest, SignatureUpdateProcessorFactoryTest, TestCSVResponseWriter, TestFieldTypeCollectionResource, IndexBasedSpellCheckerTest, JSONWriterTest, SpellPossibilityIteratorTest, TestArbitraryIndexDir, NumericFieldsTest, CopyFieldTest, BasicDistributedZkTest, ChaosMonkeySafeLeaderTest, BasicDistributedZk2Test, UnloadDistributedZkTest, OpenCloseCoreStressTest]
[junit4:junit4] Completed on J1 in 825.54s, 6 tests, 1 failure, 2 skipped <<< FAILURES!

[...truncated 407 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:388: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:877: There were test failures: 272 suites, 1143 tests, 1 error, 2 failures, 25 ignored (6 assumptions)

Total time: 99 minutes 39 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 207 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/207/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.OpenCloseCoreStressTest

Error Message:
Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=949, name=qtp1882088204-949, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest], registration stack trace below.

Stack Trace:
com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=949, name=qtp1882088204-949, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest], registration stack trace below.
        at __randomizedtesting.SeedInfo.seed([2BE095131715C45A]:0)
        at java.lang.Thread.getStackTrace(Thread.java:1495)
        at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
        at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
        at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
        at org.apache.lucene.util.LuceneTestCase.newFSDirectory(LuceneTestCase.java:929)
        at org.apache.lucene.util.LuceneTestCase.newFSDirectory(LuceneTestCase.java:909)
        at org.apache.lucene.util.LuceneTestCase.newFSDirectory(LuceneTestCase.java:904)
        at org.apache.solr.core.MockFSDirectoryFactory.create(MockFSDirectoryFactory.java:34)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:272)
        at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:224)
        at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:124)
        at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1405)
        at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1541)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:795)
        at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
        at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:927)
        at org.apache.solr.core.CoreContainer.create(CoreContainer.java:957)
        at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1158)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:226)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
        at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:365)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
        at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00016_core/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@727cc402)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
        at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        ... 1 more


FAILED:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:38767/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:16138/collection1], CloudJettyRunner [url=http://127.0.0.1:19858/collection1]]

Stack Trace:
java.lang.AssertionError: Test Setup Failure: shard1 should have just been set up to be inconsistent - but it's still consistent. Leader:http://127.0.0.1:38767/collection1skip list:[CloudJettyRunner [url=http://127.0.0.1:16138/collection1], CloudJettyRunner [url=http://127.0.0.1:19858/collection1]]
        at __randomizedtesting.SeedInfo.seed([2BE095131715C45A:AA061B0B604AA466]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertNotNull(Assert.java:526)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:213)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8513 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 0 T617 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 8 T617 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1363218059210
[junit4:junit4]   2> 10 T617 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 11 T618 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 15 T618 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 15 T618 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 15 T618 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 16 T618 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 17 T618 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-1363218059209/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 112 T617 oasc.ZkTestServer.run start zk server on port:35115
[junit4:junit4]   2> 113 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@54dc6a8
[junit4:junit4]   2> 114 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 115 T623 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 115 T623 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 116 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:57981
[junit4:junit4]   2> 117 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:57981
[junit4:junit4]   2> 117 T621 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 147 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20000 with negotiated timeout 10000 for client /140.211.11.196:57981
[junit4:junit4]   2> 147 T623 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20000, negotiated timeout = 10000
[junit4:junit4]   2> 147 T624 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@54dc6a8 name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 148 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 148 T617 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 153 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd20000
[junit4:junit4]   2> 154 T624 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 154 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd20000 closed
[junit4:junit4]   2> 156 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:57981 which had sessionid 0x13d6620cfd20000
[junit4:junit4]   2> 156 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@12116c50
[junit4:junit4]   2> 157 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 158 T625 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 158 T625 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 159 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:12321
[junit4:junit4]   2> 159 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:12321
[junit4:junit4]   2> 160 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20001 with negotiated timeout 10000 for client /140.211.11.196:12321
[junit4:junit4]   2> 161 T625 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20001, negotiated timeout = 10000
[junit4:junit4]   2> 161 T626 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12116c50 name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 162 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 162 T617 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 168 T617 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 173 T617 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 177 T617 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 184 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 185 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 200 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 202 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 210 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 211 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 218 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 219 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 225 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 226 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 232 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 233 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 240 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 241 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 246 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 247 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 253 T617 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 254 T617 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 261 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd20001
[junit4:junit4]   2> 262 T626 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 262 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:12321 which had sessionid 0x13d6620cfd20001
[junit4:junit4]   2> 262 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd20001 closed
[junit4:junit4]   2> 503 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 510 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:28426
[junit4:junit4]   2> 511 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 512 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 512 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464
[junit4:junit4]   2> 513 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/solr.xml
[junit4:junit4]   2> 513 T617 oasc.CoreContainer.<init> New CoreContainer 591092348
[junit4:junit4]   2> 514 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/'
[junit4:junit4]   2> 515 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/'
[junit4:junit4]   2> 600 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 600 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 601 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 602 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 602 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 603 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 604 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 604 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 605 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 606 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 649 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 693 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 694 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 695 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5aaa980
[junit4:junit4]   2> 697 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 698 T636 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 701 T636 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 702 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:42560
[junit4:junit4]   2> 702 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:42560
[junit4:junit4]   2> 705 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20002 with negotiated timeout 20000 for client /140.211.11.196:42560
[junit4:junit4]   2> 705 T636 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20002, negotiated timeout = 20000
[junit4:junit4]   2> 705 T637 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5aaa980 name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 706 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 709 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd20002
[junit4:junit4]   2> 711 T637 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 711 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:42560 which had sessionid 0x13d6620cfd20002
[junit4:junit4]   2> 711 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd20002 closed
[junit4:junit4]   2> 713 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 754 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3175252f
[junit4:junit4]   2> 756 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 756 T638 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 757 T638 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 757 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:17261
[junit4:junit4]   2> 758 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:17261
[junit4:junit4]   2> 760 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20003 with negotiated timeout 20000 for client /140.211.11.196:17261
[junit4:junit4]   2> 760 T638 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20003, negotiated timeout = 20000
[junit4:junit4]   2> 762 T639 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3175252f name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 762 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 767 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 771 T617 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 777 T617 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:28426_
[junit4:junit4]   2> 779 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:28426_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:28426_
[junit4:junit4]   2> 781 T617 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:28426_
[junit4:junit4]   2> 789 T617 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 804 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 807 T617 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 814 T617 oasc.Overseer.start Overseer (id=89339858728910851-127.0.0.1:28426_-n_0000000000) starting
[junit4:junit4]   2> 816 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 818 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 822 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 824 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 832 T641 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 834 T617 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 841 T617 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 847 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 857 T640 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 873 T642 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/collection1
[junit4:junit4]   2> 874 T642 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 876 T642 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 876 T642 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 882 T642 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/collection1/'
[junit4:junit4]   2> 885 T642 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/collection1/lib/README' to classloader
[junit4:junit4]   2> 885 T642 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 1039 T642 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 1183 T642 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1185 T642 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1200 T642 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 2200 T642 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 2216 T642 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 2221 T642 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 2248 T642 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2256 T642 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 2264 T642 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2267 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2267 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 2268 T642 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 2270 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 2271 T642 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 2271 T642 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2272 T642 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363218059464/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/control/data/
[junit4:junit4]   2> 2272 T642 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79024994
[junit4:junit4]   2> 2273 T642 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 2274 T642 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/control/data forceNew: false
[junit4:junit4]   2> 2274 T642 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363218059209/control/data/index/
[junit4:junit4]   2> 2275 T642 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363218059209/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 2275 T642 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/control/data/index forceNew: false
[junit4:junit4]   2> 2281 T642 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3aea91be lockFactory=org.apache.lucene.store.NativeFSLockFactory@586c2d62),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 2281 T642 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 2285 T642 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 2285 T642 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 2286 T642 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 2287 T642 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 2288 T642 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 2288 T642 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 2289 T642 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 2290 T642 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 2290 T642 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 2303 T642 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 2313 T642 oass.SolrIndexSearcher.<init> Opening Searcher@57410e49 main
[junit4:junit4]   2> 2314 T642 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/control/data/tlog
[junit4:junit4]   2> 2315 T642 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 2315 T642 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 2321 T643 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@57410e49 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 2325 T642 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 2325 T642 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 2326 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2410 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 2411 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:28426_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:28426"}
[junit4:junit4]   2> 2412 T640 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2412 T640 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2413 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2435 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3409 T642 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 3409 T642 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:28426 collection:control_collection shard:shard1
[junit4:junit4]   2> 3411 T642 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 3455 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 3457 T642 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 3458 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3469 T642 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 3469 T642 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 3470 T642 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:28426/collection1/
[junit4:junit4]   2> 3470 T642 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 3470 T642 oasc.SyncStrategy.syncToMe http://127.0.0.1:28426/collection1/ has no replicas
[junit4:junit4]   2> 3470 T642 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:28426/collection1/
[junit4:junit4]   2> 3471 T642 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 3477 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3945 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 3966 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 4003 T642 oasc.ZkController.register We are http://127.0.0.1:28426/collection1/ and leader is http://127.0.0.1:28426/collection1/
[junit4:junit4]   2> 4003 T642 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:28426
[junit4:junit4]   2> 4003 T642 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 4004 T642 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 4004 T642 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 4007 T642 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 4008 T617 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 4009 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 4010 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 4031 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 4033 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@18369f6d
[junit4:junit4]   2> 4034 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4035 T645 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4035 T645 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 4036 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:60081
[junit4:junit4]   2> 4036 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:60081
[junit4:junit4]   2> 4038 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20004 with negotiated timeout 10000 for client /140.211.11.196:60081
[junit4:junit4]   2> 4038 T645 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20004, negotiated timeout = 10000
[junit4:junit4]   2> 4038 T646 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18369f6d name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4039 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4041 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 4045 T617 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4283 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4286 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38368
[junit4:junit4]   2> 4287 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4288 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4288 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245
[junit4:junit4]   2> 4289 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/solr.xml
[junit4:junit4]   2> 4289 T617 oasc.CoreContainer.<init> New CoreContainer 494633917
[junit4:junit4]   2> 4290 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/'
[junit4:junit4]   2> 4291 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/'
[junit4:junit4]   2> 4352 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4353 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4353 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4354 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4354 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4355 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4355 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4356 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4357 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4358 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4385 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4413 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 4414 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4414 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@55bf573b
[junit4:junit4]   2> 4415 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4416 T656 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4417 T656 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 4417 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:51541
[junit4:junit4]   2> 4417 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:51541
[junit4:junit4]   2> 4420 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20005 with negotiated timeout 20000 for client /140.211.11.196:51541
[junit4:junit4]   2> 4420 T656 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20005, negotiated timeout = 20000
[junit4:junit4]   2> 4420 T657 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@55bf573b name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4421 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4423 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd20005
[junit4:junit4]   2> 4432 T657 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4432 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:51541 which had sessionid 0x13d6620cfd20005
[junit4:junit4]   2> 4432 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd20005 closed
[junit4:junit4]   2> 4434 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4446 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3e75a7f3
[junit4:junit4]   2> 4448 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4448 T658 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4449 T658 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 4449 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:10294
[junit4:junit4]   2> 4450 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:10294
[junit4:junit4]   2> 4452 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20006 with negotiated timeout 20000 for client /140.211.11.196:10294
[junit4:junit4]   2> 4452 T658 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20006, negotiated timeout = 20000
[junit4:junit4]   2> 4452 T659 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e75a7f3 name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4452 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4454 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4456 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4459 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 5465 T617 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38368_
[junit4:junit4]   2> 5466 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:38368_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:38368_
[junit4:junit4]   2> 5494 T617 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38368_
[junit4:junit4]   2> 5496 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 5497 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:28426_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:28426"}
[junit4:junit4]   2> 5499 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 5509 T659 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5509 T646 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5524 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5524 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5528 T660 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/collection1
[junit4:junit4]   2> 5528 T660 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5529 T660 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5529 T660 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5530 T639 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5530 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5531 T660 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/collection1/'
[junit4:junit4]   2> 5531 T660 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/collection1/lib/README' to classloader
[junit4:junit4]   2> 5532 T660 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5575 T660 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 5641 T660 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5643 T660 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5654 T660 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6442 T660 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6459 T660 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6465 T660 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6493 T660 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6500 T660 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6508 T660 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6510 T660 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6511 T660 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6511 T660 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6513 T660 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6514 T660 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6514 T660 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6514 T660 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363218063245/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty1/
[junit4:junit4]   2> 6515 T660 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79024994
[junit4:junit4]   2> 6516 T660 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6516 T660 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty1 forceNew: false
[junit4:junit4]   2> 6517 T660 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty1/index/
[junit4:junit4]   2> 6517 T660 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6518 T660 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty1/index forceNew: false
[junit4:junit4]   2> 6521 T660 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6522 T660 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6526 T660 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6526 T660 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6527 T660 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6528 T660 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6529 T660 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6530 T660 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6530 T660 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6531 T660 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6532 T660 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6542 T660 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6550 T660 oass.SolrIndexSearcher.<init> Opening Searcher@1d727f16 main
[junit4:junit4]   2> 6550 T660 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty1/tlog
[junit4:junit4]   2> 6552 T660 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6552 T660 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6559 T661 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1d727f16 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6563 T660 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6563 T660 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 7033 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 7034 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:38368_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:38368"}
[junit4:junit4]   2> 7034 T640 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 7034 T640 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 7065 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7065 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7065 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7594 T660 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7594 T660 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38368 collection:collection1 shard:shard1
[junit4:junit4]   2> 7595 T660 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7626 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7632 T660 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7633 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7637 T660 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7637 T660 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7637 T660 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 7638 T660 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7638 T660 oasc.SyncStrategy.syncToMe http://127.0.0.1:38368/collection1/ has no replicas
[junit4:junit4]   2> 7639 T660 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 7639 T660 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7647 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8575 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8632 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8632 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8632 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8678 T660 oasc.ZkController.register We are http://127.0.0.1:38368/collection1/ and leader is http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 8678 T660 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38368
[junit4:junit4]   2> 8678 T660 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8678 T660 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8679 T660 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8681 T660 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8682 T617 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8683 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8684 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8932 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8936 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:38767
[junit4:junit4]   2> 8937 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8938 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8939 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905
[junit4:junit4]   2> 8939 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/solr.xml
[junit4:junit4]   2> 8940 T617 oasc.CoreContainer.<init> New CoreContainer 1732922169
[junit4:junit4]   2> 8940 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/'
[junit4:junit4]   2> 8941 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/'
[junit4:junit4]   2> 9001 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 9002 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 9003 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 9003 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 9004 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 9004 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 9005 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 9005 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 9006 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 9007 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 9034 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 9062 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 9063 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 9063 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3663c64f
[junit4:junit4]   2> 9065 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 9065 T672 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 9067 T672 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 9067 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:23847
[junit4:junit4]   2> 9067 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:23847
[junit4:junit4]   2> 9070 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20007 with negotiated timeout 20000 for client /140.211.11.196:23847
[junit4:junit4]   2> 9070 T672 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20007, negotiated timeout = 20000
[junit4:junit4]   2> 9070 T673 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3663c64f name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 9070 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 9072 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd20007
[junit4:junit4]   2> 9085 T673 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 9085 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:23847 which had sessionid 0x13d6620cfd20007
[junit4:junit4]   2> 9085 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd20007 closed
[junit4:junit4]   2> 9086 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 9094 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@45f0068e
[junit4:junit4]   2> 9095 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 9095 T674 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 9096 T674 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 9096 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:11432
[junit4:junit4]   2> 9096 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:11432
[junit4:junit4]   2> 9098 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20008 with negotiated timeout 20000 for client /140.211.11.196:11432
[junit4:junit4]   2> 9098 T674 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20008, negotiated timeout = 20000
[junit4:junit4]   2> 9098 T675 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45f0068e name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 9098 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 9100 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9102 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 9105 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 10111 T617 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38767_
[junit4:junit4]   2> 10112 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd20008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:38767_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:38767_
[junit4:junit4]   2> 10140 T617 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38767_
[junit4:junit4]   2> 10144 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 10144 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 10145 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 10145 T659 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10146 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10146 T675 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10146 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:38368_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:38368"}
[junit4:junit4]   2> 10153 T646 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10162 T676 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/collection1
[junit4:junit4]   2> 10163 T676 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 10163 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10163 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10163 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10164 T676 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 10165 T676 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 10168 T676 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/collection1/'
[junit4:junit4]   2> 10170 T676 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 10171 T676 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/collection1/lib/README' to classloader
[junit4:junit4]   2> 10171 T639 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 10171 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10231 T676 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 10305 T676 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10306 T676 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10316 T676 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 11330 T676 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 11352 T676 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 11358 T676 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 11415 T676 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 11425 T676 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 11433 T676 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 11436 T676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 11437 T676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 11437 T676 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 11439 T676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 11439 T676 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 11440 T676 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 11440 T676 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363218067905/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2/
[junit4:junit4]   2> 11441 T676 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79024994
[junit4:junit4]   2> 11441 T676 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 11443 T676 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2 forceNew: false
[junit4:junit4]   2> 11443 T676 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2/index/
[junit4:junit4]   2> 11443 T676 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 11444 T676 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2/index forceNew: false
[junit4:junit4]   2> 11449 T676 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7f6578c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@307cffec),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 11449 T676 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 11453 T676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 11454 T676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 11455 T676 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 11456 T676 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 11457 T676 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 11457 T676 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 11458 T676 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 11459 T676 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 11460 T676 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 11477 T676 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 11490 T676 oass.SolrIndexSearcher.<init> Opening Searcher@6522008b main
[junit4:junit4]   2> 11490 T676 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2/tlog
[junit4:junit4]   2> 11491 T676 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 11492 T676 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 11522 T677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6522008b main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 11527 T676 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 11527 T676 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11671 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 11673 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:38767_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:38767"}
[junit4:junit4]   2> 11673 T640 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 11673 T640 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 11683 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11683 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11683 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11684 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 12531 T676 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 12531 T676 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38767 collection:collection1 shard:shard1
[junit4:junit4]   2> 12557 T676 oasc.ZkController.register We are http://127.0.0.1:38767/collection1/ and leader is http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 12558 T676 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38767
[junit4:junit4]   2> 12558 T676 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 12558 T676 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C21 name=collection1 org.apache.solr.core.SolrCore@8f7ad17 url=http://127.0.0.1:38767/collection1 node=127.0.0.1:38767_ C21_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:38767_, base_url=http://127.0.0.1:38767}
[junit4:junit4]   2> 12559 T678 C21 P38767 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 12561 T676 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 12561 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 12561 T678 C21 P38767 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 12562 T678 C21 P38767 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 12564 T617 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 12565 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 12566 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12566 T678 C21 P38767 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12891 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 12895 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:29531
[junit4:junit4]   2> 12896 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 12897 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 12898 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799
[junit4:junit4]   2> 12898 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/solr.xml
[junit4:junit4]   2> 12899 T617 oasc.CoreContainer.<init> New CoreContainer 343386315
[junit4:junit4]   2> 12900 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/'
[junit4:junit4]   2> 12901 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/'
[junit4:junit4]   2> 12977 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 12978 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 12978 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 12979 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 12980 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 12980 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 12981 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 12982 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 12982 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 12983 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13016 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13043 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 13044 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13044 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2d471baf
[junit4:junit4]   2> 13046 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13046 T689 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13047 T689 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 13048 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:58024
[junit4:junit4]   2> 13048 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:58024
[junit4:junit4]   2> 13051 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20009 with negotiated timeout 20000 for client /140.211.11.196:58024
[junit4:junit4]   2> 13051 T689 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd20009, negotiated timeout = 20000
[junit4:junit4]   2> 13052 T690 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2d471baf name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13052 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13054 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd20009
[junit4:junit4]   2> 13057 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:58024 which had sessionid 0x13d6620cfd20009
[junit4:junit4]   2> 13057 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd20009 closed
[junit4:junit4]   2> 13057 T690 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13058 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13078 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d0840da
[junit4:junit4]   2> 13080 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13081 T691 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13082 T691 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 13082 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:28892
[junit4:junit4]   2> 13083 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:28892
[junit4:junit4]   2> 13085 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd2000a with negotiated timeout 20000 for client /140.211.11.196:28892
[junit4:junit4]   2> 13085 T691 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd2000a, negotiated timeout = 20000
[junit4:junit4]   2> 13085 T692 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d0840da name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13086 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13090 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000a type:create cxid:0x1 zxid:0x81 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13092 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000a type:create cxid:0x2 zxid:0x82 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13098 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 13196 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 13198 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:38767_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:38767"}
[junit4:junit4]   2> 13212 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13212 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13212 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13212 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13213 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13602 T649 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=3&state=recovering&nodeName=127.0.0.1:38767_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 14104 T617 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:29531_
[junit4:junit4]   2> 14105 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000a type:delete cxid:0xd zxid:0x87 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:29531_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:29531_
[junit4:junit4]   2> 14138 T617 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:29531_
[junit4:junit4]   2> 14142 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14142 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14143 T659 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14143 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14143 T675 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14143 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14144 T692 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14144 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14145 T639 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14146 T646 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14161 T693 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/collection1
[junit4:junit4]   2> 14162 T693 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14163 T693 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14163 T693 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14166 T693 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/collection1/'
[junit4:junit4]   2> 14168 T693 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/collection1/lib/README' to classloader
[junit4:junit4]   2> 14169 T693 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14237 T693 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 14308 T693 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14310 T693 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14319 T693 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 15166 T693 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15186 T693 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15193 T693 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15220 T693 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15228 T693 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15236 T693 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15238 T693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15238 T693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15239 T693 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15241 T693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15241 T693 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15241 T693 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15242 T693 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363218071799/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3/
[junit4:junit4]   2> 15242 T693 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79024994
[junit4:junit4]   2> 15243 T693 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15244 T693 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3 forceNew: false
[junit4:junit4]   2> 15244 T693 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3/index/
[junit4:junit4]   2> 15245 T693 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 15245 T693 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3/index forceNew: false
[junit4:junit4]   2> 15250 T693 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5463f35f lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf5ed2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15250 T693 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15255 T693 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 15255 T693 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 15256 T693 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 15257 T693 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 15258 T693 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 15259 T693 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 15259 T693 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 15260 T693 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 15261 T693 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 15273 T693 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 15282 T693 oass.SolrIndexSearcher.<init> Opening Searcher@27e67ef3 main
[junit4:junit4]   2> 15283 T693 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3/tlog
[junit4:junit4]   2> 15284 T693 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 15285 T693 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 15292 T694 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@27e67ef3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 15295 T693 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15295 T693 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2>  C21_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:38767_, base_url=http://127.0.0.1:38767}
[junit4:junit4]   2> 15604 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38368/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 15604 T678 C21 P38767 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:38767 START replicas=[http://127.0.0.1:38368/collection1/] nUpdates=100
[junit4:junit4]   2> 15605 T678 C21 P38767 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 15606 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 15607 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 15607 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 15608 T678 C21 P38767 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 15608 T678 C21 P38767 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38368/collection1/. core=collection1
[junit4:junit4]   2> 15608 T678 C21 P38767 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C22 name=collection1 org.apache.solr.core.SolrCore@2b663f09 url=http://127.0.0.1:38368/collection1 node=127.0.0.1:38368_ C22_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:38368_, base_url=http://127.0.0.1:38368, leader=true}
[junit4:junit4]   2> 15617 T650 C22 P38368 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 15641 T651 C22 P38368 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15646 T651 C22 P38368 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15647 T651 C22 P38368 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15648 T651 C22 P38368 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15648 T651 C22 P38368 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15649 T651 C22 P38368 oass.SolrIndexSearcher.<init> Opening Searcher@4062ce9b realtime
[junit4:junit4]   2> 15650 T651 C22 P38368 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15650 T651 C22 P38368 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 15652 T678 C21 P38767 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15652 T678 C21 P38767 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 15696 T652 C22 P38368 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15697 T652 C22 P38368 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4]   2> 15699 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 15699 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 15699 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 15703 T652 C22 P38368 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4]   2> 15704 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 15706 T678 C21 P38767 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty2/index.20130313234114905 forceNew: false
[junit4:junit4]   2> 15706 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@69a90966 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d6ced76) fullCopy=false
[junit4:junit4]   2> 15710 T652 C22 P38368 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4]   2> 15711 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15713 T678 C21 P38767 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15714 T678 C21 P38767 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 15728 T678 C21 P38767 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7f6578c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@307cffec),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7f6578c4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@307cffec),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15729 T678 C21 P38767 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15729 T678 C21 P38767 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 15730 T678 C21 P38767 oass.SolrIndexSearcher.<init> Opening Searcher@30d4c046 main
[junit4:junit4]   2> 15732 T677 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@30d4c046 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 15733 T678 C21 P38767 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@69a90966 lockFactory=org.apache.lucene.store.NativeFSLockFactory@d6ced76)
[junit4:junit4]   2> 15733 T678 C21 P38767 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 15733 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 15733 T678 C21 P38767 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 15734 T678 C21 P38767 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15878 T678 C21 P38767 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 16228 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 16229 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:29531_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:29531"}
[junit4:junit4]   2> 16230 T640 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 16230 T640 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16244 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:38767_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:38767"}
[junit4:junit4]   2> 16257 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16257 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16258 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16257 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16258 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16299 T693 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16299 T693 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:29531 collection:collection1 shard:shard1
[junit4:junit4]   2> 16308 T693 oasc.ZkController.register We are http://127.0.0.1:29531/collection1/ and leader is http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 16308 T693 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:29531
[junit4:junit4]   2> 16309 T693 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16309 T693 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C23 name=collection1 org.apache.solr.core.SolrCore@2b027c27 url=http://127.0.0.1:29531/collection1 node=127.0.0.1:29531_ C23_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:29531_, base_url=http://127.0.0.1:29531}
[junit4:junit4]   2> 16309 T697 C23 P29531 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16310 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16311 T697 C23 P29531 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16311 T693 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 16311 T697 C23 P29531 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16314 T617 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 16315 T697 C23 P29531 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16315 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16316 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16684 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16687 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16138
[junit4:junit4]   2> 16688 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16689 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16690 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537
[junit4:junit4]   2> 16690 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/solr.xml
[junit4:junit4]   2> 16691 T617 oasc.CoreContainer.<init> New CoreContainer 627422048
[junit4:junit4]   2> 16692 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/'
[junit4:junit4]   2> 16693 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/'
[junit4:junit4]   2> 16768 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 16769 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 16769 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 16770 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 16771 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 16771 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 16772 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 16773 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 16773 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 16774 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 16815 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 16850 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 16851 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 16852 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@368ceb0c
[junit4:junit4]   2> 16854 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16856 T708 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16857 T708 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 16858 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:32619
[junit4:junit4]   2> 16858 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:32619
[junit4:junit4]   2> 16870 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd2000b with negotiated timeout 20000 for client /140.211.11.196:32619
[junit4:junit4]   2> 16872 T708 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd2000b, negotiated timeout = 20000
[junit4:junit4]   2> 16873 T709 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@368ceb0c name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16873 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16876 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd2000b
[junit4:junit4]   2> 16878 T709 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 16879 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:32619 which had sessionid 0x13d6620cfd2000b
[junit4:junit4]   2> 16878 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd2000b closed
[junit4:junit4]   2> 16881 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 16898 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@106d908c
[junit4:junit4]   2> 16901 T710 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16902 T710 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 16902 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16902 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:49222
[junit4:junit4]   2> 16903 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:49222
[junit4:junit4]   2> 16906 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd2000c with negotiated timeout 20000 for client /140.211.11.196:49222
[junit4:junit4]   2> 16906 T710 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd2000c, negotiated timeout = 20000
[junit4:junit4]   2> 16907 T711 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@106d908c name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16907 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16909 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000c type:create cxid:0x1 zxid:0x98 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16918 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000c type:create cxid:0x2 zxid:0x99 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16945 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 17772 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 17773 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:29531_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:29531"}
[junit4:junit4]   2> 17784 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17784 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17784 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17784 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17785 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17784 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17951 T617 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16138_
[junit4:junit4]   2> 17953 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000c type:delete cxid:0xd zxid:0x9e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:16138_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:16138_
[junit4:junit4]   2> 17955 T617 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16138_
[junit4:junit4]   2> 17960 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17960 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17961 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17961 T659 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17961 T675 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17961 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17961 T692 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17961 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17962 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17964 T639 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17964 T711 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17964 T646 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17973 T712 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/collection1
[junit4:junit4]   2> 17973 T712 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 17975 T712 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 17975 T712 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 17978 T712 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/collection1/'
[junit4:junit4]   2> 17979 T712 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 17980 T712 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/collection1/lib/README' to classloader
[junit4:junit4]   2> 18046 T712 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 18126 T712 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 18128 T712 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 18139 T712 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 18338 T653 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=4&state=recovering&nodeName=127.0.0.1:29531_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4]   2> 19021 T712 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 19040 T712 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 19046 T712 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 19074 T712 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19082 T712 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 19089 T712 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19093 T712 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19093 T712 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19094 T712 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 19096 T712 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 19096 T712 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 19096 T712 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19097 T712 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363218075537/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4/
[junit4:junit4]   2> 19097 T712 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79024994
[junit4:junit4]   2> 19098 T712 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 19102 T712 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4 forceNew: false
[junit4:junit4]   2> 19102 T712 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4/index/
[junit4:junit4]   2> 19102 T712 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 19103 T712 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4/index forceNew: false
[junit4:junit4]   2> 19107 T712 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5acc4401 lockFactory=org.apache.lucene.store.NativeFSLockFactory@367bbec2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 19108 T712 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 19111 T712 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 19112 T712 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 19113 T712 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 19114 T712 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 19115 T712 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 19115 T712 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 19116 T712 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 19117 T712 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 19117 T712 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 19128 T712 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 19138 T712 oass.SolrIndexSearcher.<init> Opening Searcher@5683b55e main
[junit4:junit4]   2> 19138 T712 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4/tlog
[junit4:junit4]   2> 19140 T712 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 19140 T712 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 19148 T713 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5683b55e main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 19152 T712 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 19153 T712 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19301 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 19302 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:16138_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:16138"}
[junit4:junit4]   2> 19302 T640 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 19303 T640 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 19340 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19340 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19340 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19340 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19340 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19340 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20184 T712 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 20184 T712 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16138 collection:collection1 shard:shard1
[junit4:junit4]   2> 20194 T712 oasc.ZkController.register We are http://127.0.0.1:16138/collection1/ and leader is http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 20194 T712 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16138
[junit4:junit4]   2> 20194 T712 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 20195 T712 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@12065f1a url=http://127.0.0.1:16138/collection1 node=127.0.0.1:16138_ C24_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:16138_, base_url=http://127.0.0.1:16138}
[junit4:junit4]   2> 20195 T714 C24 P16138 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 20197 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 20197 T714 C24 P16138 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 20197 T712 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 20197 T714 C24 P16138 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20199 T617 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 20200 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 20200 T714 C24 P16138 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20201 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2>  C23_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:29531_, base_url=http://127.0.0.1:29531}
[junit4:junit4]   2> 20340 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38368/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 20340 T697 C23 P29531 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:29531 START replicas=[http://127.0.0.1:38368/collection1/] nUpdates=100
[junit4:junit4]   2> 20341 T697 C23 P29531 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 20342 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 20342 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 20343 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 20343 T697 C23 P29531 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 20343 T697 C23 P29531 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38368/collection1/. core=collection1
[junit4:junit4]   2> 20344 T697 C23 P29531 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20345 T650 C22 P38368 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 20366 T655 C22 P38368 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 20368 T655 C22 P38368 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 20369 T655 C22 P38368 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 20370 T655 C22 P38368 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 20370 T655 C22 P38368 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 20372 T697 C23 P29531 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20372 T697 C23 P29531 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 20376 T652 C22 P38368 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4]   2> 20377 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 20377 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 20378 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 20381 T652 C22 P38368 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=1
[junit4:junit4]   2> 20382 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 20384 T697 C23 P29531 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty3/index.20130313234119583 forceNew: false
[junit4:junit4]   2> 20384 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@64dd7a0e lockFactory=org.apache.lucene.store.NativeFSLockFactory@171f932a) fullCopy=false
[junit4:junit4]   2> 20388 T652 C22 P38368 REQ /replication {file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=1
[junit4:junit4]   2> 20390 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 20392 T697 C23 P29531 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 20392 T697 C23 P29531 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 20395 T697 C23 P29531 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5463f35f lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf5ed2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5463f35f lockFactory=org.apache.lucene.store.NativeFSLockFactory@3bf5ed2),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 20396 T697 C23 P29531 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 20396 T697 C23 P29531 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 20396 T697 C23 P29531 oass.SolrIndexSearcher.<init> Opening Searcher@1541703c main
[junit4:junit4]   2> 20398 T694 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1541703c main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 20399 T697 C23 P29531 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@64dd7a0e lockFactory=org.apache.lucene.store.NativeFSLockFactory@171f932a)
[junit4:junit4]   2> 20399 T697 C23 P29531 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 20400 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 20400 T697 C23 P29531 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 20400 T697 C23 P29531 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20403 T697 C23 P29531 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20525 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 20528 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:49323
[junit4:junit4]   2> 20529 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 20529 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 20530 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431
[junit4:junit4]   2> 20530 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/solr.xml
[junit4:junit4]   2> 20531 T617 oasc.CoreContainer.<init> New CoreContainer 960107123
[junit4:junit4]   2> 20532 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/'
[junit4:junit4]   2> 20532 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/'
[junit4:junit4]   2> 20602 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 20603 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 20603 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 20604 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 20605 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 20605 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 20606 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 20607 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 20607 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 20608 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 20646 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 20677 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 20678 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 20678 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2b827602
[junit4:junit4]   2> 20680 T727 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 20680 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 20682 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:28640
[junit4:junit4]   2> 20681 T727 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 20683 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:28640
[junit4:junit4]   2> 20685 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd2000d with negotiated timeout 20000 for client /140.211.11.196:28640
[junit4:junit4]   2> 20685 T727 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd2000d, negotiated timeout = 20000
[junit4:junit4]   2> 20686 T728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2b827602 name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 20686 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 20688 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd2000d
[junit4:junit4]   2> 20690 T728 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 20690 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:28640 which had sessionid 0x13d6620cfd2000d
[junit4:junit4]   2> 20690 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd2000d closed
[junit4:junit4]   2> 20691 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 20713 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@4e8659e9
[junit4:junit4]   2> 20715 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 20715 T729 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 20716 T729 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 20716 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:15895
[junit4:junit4]   2> 20717 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:15895
[junit4:junit4]   2> 20719 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd2000e with negotiated timeout 20000 for client /140.211.11.196:15895
[junit4:junit4]   2> 20720 T729 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd2000e, negotiated timeout = 20000
[junit4:junit4]   2> 20720 T730 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e8659e9 name:ZooKeeperConnection Watcher:127.0.0.1:35115/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 20720 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 20723 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000e type:create cxid:0x1 zxid:0xac txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 20725 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000e type:create cxid:0x2 zxid:0xad txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 20729 T617 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 20853 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 20856 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:16138_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:16138"}
[junit4:junit4]   2> 20902 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:29531_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:29531"}
[junit4:junit4]   2> 20913 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20913 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20913 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20913 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20913 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20913 T730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20913 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21236 T654 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=5&state=recovering&nodeName=127.0.0.1:16138_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 21737 T617 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49323_
[junit4:junit4]   2> 21738 T622 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6620cfd2000e type:delete cxid:0xd zxid:0xb5 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:49323_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:49323_
[junit4:junit4]   2> 21757 T617 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49323_
[junit4:junit4]   2> 21809 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21809 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21809 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21810 T659 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21811 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 21811 T692 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21811 T730 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21811 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 21812 T675 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21812 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 21813 T730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 21813 T646 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21814 T639 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21813 T711 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 21844 T731 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/collection1
[junit4:junit4]   2> 21845 T731 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 21846 T731 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 21846 T731 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 21849 T731 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/collection1/'
[junit4:junit4]   2> 21851 T731 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 21853 T731 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/collection1/lib/README' to classloader
[junit4:junit4]   2> 21982 T731 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 22079 T731 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 22080 T731 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 22093 T731 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 23031 T731 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 23047 T731 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 23052 T731 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 23077 T731 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 23083 T731 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 23088 T731 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 23089 T731 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 23090 T731 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 23090 T731 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 23091 T731 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 23091 T731 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 23092 T731 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 23092 T731 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363218079431/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty5/
[junit4:junit4]   2> 23092 T731 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@79024994
[junit4:junit4]   2> 23093 T731 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 23093 T731 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty5 forceNew: false
[junit4:junit4]   2> 23094 T731 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty5/index/
[junit4:junit4]   2> 23094 T731 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 23094 T731 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty5/index forceNew: false
[junit4:junit4]   2> 23097 T731 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@27d59293 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1f63a81e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 23097 T731 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 23100 T731 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 23100 T731 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 23101 T731 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 23101 T731 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 23102 T731 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 23102 T731 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 23102 T731 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 23103 T731 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 23103 T731 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 23110 T731 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 23115 T731 oass.SolrIndexSearcher.<init> Opening Searcher@60c6fe68 main
[junit4:junit4]   2> 23116 T731 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty5/tlog
[junit4:junit4]   2> 23117 T731 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 23117 T731 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 23121 T732 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@60c6fe68 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 23124 T731 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 23124 T731 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2>  C24_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:16138_, base_url=http://127.0.0.1:16138}
[junit4:junit4]   2> 23239 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:38368/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 23239 T714 C24 P16138 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:16138 START replicas=[http://127.0.0.1:38368/collection1/] nUpdates=100
[junit4:junit4]   2> 23241 T714 C24 P16138 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 23242 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 23242 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 23242 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 23243 T714 C24 P16138 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 23243 T714 C24 P16138 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:38368/collection1/. core=collection1
[junit4:junit4]   2> 23243 T714 C24 P16138 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 23244 T650 C22 P38368 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 23270 T649 C22 P38368 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23271 T649 C22 P38368 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=3
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2cd39e6b lockFactory=org.apache.lucene.store.NativeFSLockFactory@4873daf9),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 23272 T649 C22 P38368 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 23272 T649 C22 P38368 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23273 T649 C22 P38368 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 23274 T714 C24 P16138 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23274 T714 C24 P16138 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 23277 T652 C22 P38368 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 23278 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 23279 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 23279 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 23281 T652 C22 P38368 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0
[junit4:junit4]   2> 23314 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 23316 T714 C24 P16138 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363218059209/jetty4/index.20130313234122514 forceNew: false
[junit4:junit4]   2> 23316 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4269b79 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f438ccd) fullCopy=false
[junit4:junit4]   2> 23320 T652 C22 P38368 REQ /replication {file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4]   2> 23322 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 23324 T714 C24 P16138 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 23325 T714 C24 P16138 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 23327 T714 C24 P16138 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5acc4401 lockFactory=org.apache.lucene.store.NativeFSLockFactory@367bbec2),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5acc4401 lockFactory=org.apache.lucene.store.NativeFSLockFactory@367bbec2),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 23328 T714 C24 P16138 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 23328 T714 C24 P16138 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 23329 T714 C24 P16138 oass.SolrIndexSearcher.<init> Opening Searcher@1f2e3a27 main
[junit4:junit4]   2> 23330 T713 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f2e3a27 main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 23331 T714 C24 P16138 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@4269b79 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2f438ccd)
[junit4:junit4]   2> 23331 T714 C24 P16138 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 23332 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 23332 T714 C24 P16138 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 23332 T714 C24 P16138 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23354 T714 C24 P16138 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 23930 T640 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23931 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"6",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:49323_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:49323"}
[junit4:junit4]   2> 23932 T640 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 23932 T640 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 23937 T640 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:16138_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:16138"}
[junit4:junit4]   2> 23953 T639 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23953 T659 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23954 T692 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23954 T646 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23954 T675 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23953 T711 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23953 T730 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 24154 T731 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 24154 T731 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49323 collection:collection1 shard:shard1
[junit4:junit4]   2> 24162 T731 oasc.ZkController.register We are http://127.0.0.1:49323/collection1/ and leader is http://127.0.0.1:38368/collection1/
[junit4:junit4]   2> 24162 T731 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49323
[junit4:junit4]   2> 24162 T731 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 24163 T731 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@2405215f url=http://127.0.0.1:49323/collection1 node=127.0.0.1:49323_ C25_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:49323_, base_url=http://127.0.0.1:49323}
[junit4:junit4]   2> 24163 T735 C25 P49323 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 24164 T735 C25 P49323 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 24165 T735 C25 P49323 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 24165 T731 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 24165 T735 C25 P49323 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 24167 T617 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 24168 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 24169 T735 C25 P49323 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 24168 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 24452 T617 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 24456 T617 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:51772
[junit4:junit4]   2> 24457 T617 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 24458 T617 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 24459 T617 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363218083388
[junit4:junit4]   2> 24459 T617 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363218083388/solr.xml
[junit4:junit4]   2> 24460 T617 oasc.CoreContainer.<init> New CoreContainer 1730495579
[junit4:junit4]   2> 24461 T617 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363218083388/'
[junit4:junit4]   2> 24462 T617 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363218083388/'
[junit4:junit4]   2> 24543 T617 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 24544 T617 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 24545 T617 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 24545 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 24546 T617 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 24547 T617 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 24548 T617 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 24548 T617 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 24549 T617 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 24550 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 24587 T617 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 24618 T617 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:35115/solr
[junit4:junit4]   2> 24619 T617 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 24620 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@6d354203
[junit4:junit4]   2> 24622 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 24622 T746 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 24623 T746 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 24623 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:16612
[junit4:junit4]   2> 24624 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:16612
[junit4:junit4]   2> 24626 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd2000f with negotiated timeout 20000 for client /140.211.11.196:16612
[junit4:junit4]   2> 24626 T746 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:35115, sessionid = 0x13d6620cfd2000f, negotiated timeout = 20000
[junit4:junit4]   2> 24627 T747 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d354203 name:ZooKeeperConnection Watcher:127.0.0.1:35115 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 24627 T617 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 24629 T622 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6620cfd2000f
[junit4:junit4]   2> 24632 T747 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 24632 T619 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:16612 which had sessionid 0x13d6620cfd2000f
[junit4:junit4]   2> 24632 T617 oaz.ZooKeeper.close Session: 0x13d6620cfd2000f closed
[junit4:junit4]   2> 24633 T617 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 24653 T617 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:35115/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@b9ad2d4
[junit4:junit4]   2> 24655 T617 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 24655 T748 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:35115. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 24656 T748 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:35115, initiating session
[junit4:junit4]   2> 24656 T619 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:10242
[junit4:junit4]   2> 24657 T619 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:10242
[junit4:junit4]   2> 24659 T621 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6620cfd20010 with negotiated timeout 20000 for client /140.211.11.196:10242
[junit4:junit4]   2> 246

[...truncated too long message...]

junit4]   2> 785703 T334 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data
[junit4:junit4]   2> 785704 T334 oasc.CachingDirectoryFactory.close Closing directory when closing factory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data/index
[junit4:junit4]   2> 785704 T334 oasc.CachingDirectoryFactory.closeDirectory Closing directory: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00019_core/data/index
[junit4:junit4]   2> 785749 T334 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
[junit4:junit4]   2> 786357 T334 oas.SolrTestCaseJ4.tearDown ###Ending test10MinutesNew
[junit4:junit4]   2> 786359 T334 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> NOTE: test params are: codec=Asserting, sim=RandomSimilarityProvider(queryNorm=false,coord=yes): {}, locale=el_GR, timezone=Australia/Lord_Howe
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=33589200,total=532873216
[junit4:junit4]   2> NOTE: All tests run in this JVM: [IndexBasedSpellCheckerTest, NoCacheHeaderTest, TestLMJelinekMercerSimilarityFactory, SortByFunctionTest, TestMultiCoreConfBootstrap, TestDefaultSimilarityFactory, UniqFieldsUpdateProcessorFactoryTest, TestLazyCores, DisMaxRequestHandlerTest, TestCollationKeyRangeQueries, DefaultValueUpdateProcessorTest, SolrPluginUtilsTest, LeaderElectionIntegrationTest, TestRecovery, RequestHandlersTest, TestCodecSupport, CurrencyFieldOpenExchangeTest, OpenCloseCoreStressTest]
[junit4:junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=OpenCloseCoreStressTest -Dtests.seed=2BE095131715C45A -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=el_GR -Dtests.timezone=Australia/Lord_Howe -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR   0.00s J1 | OpenCloseCoreStressTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ResourceDisposalError: Resource in scope SUITE failed to close. Resource was registered from thread Thread[id=949, name=qtp1882088204-949, state=RUNNABLE, group=TGRP-OpenCloseCoreStressTest], registration stack trace below.
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([2BE095131715C45A]:0)
[junit4:junit4]    > at java.lang.Thread.getStackTrace(Thread.java:1495)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedContext.closeAtEnd(RandomizedContext.java:150)
[junit4:junit4]    > at org.apache.lucene.util.LuceneTestCase.closeAfterSuite(LuceneTestCase.java:520)
[junit4:junit4]    > at org.apache.lucene.util.LuceneTestCase.wrapDirectory(LuceneTestCase.java:986)
[junit4:junit4]    > at org.apache.lucene.util.LuceneTestCase.newFSDirectory(LuceneTestCase.java:929)
[junit4:junit4]    > at org.apache.lucene.util.LuceneTestCase.newFSDirectory(LuceneTestCase.java:909)
[junit4:junit4]    > at org.apache.lucene.util.LuceneTestCase.newFSDirectory(LuceneTestCase.java:904)
[junit4:junit4]    > at org.apache.solr.core.MockFSDirectoryFactory.create(MockFSDirectoryFactory.java:34)
[junit4:junit4]    > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:272)
[junit4:junit4]    > at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:224)
[junit4:junit4]    > at org.apache.solr.search.SolrIndexSearcher.<init>(SolrIndexSearcher.java:124)
[junit4:junit4]    > at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1405)
[junit4:junit4]    > at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1541)
[junit4:junit4]    > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:795)
[junit4:junit4]    > at org.apache.solr.core.SolrCore.<init>(SolrCore.java:619)
[junit4:junit4]    > at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:927)
[junit4:junit4]    > at org.apache.solr.core.CoreContainer.create(CoreContainer.java:957)
[junit4:junit4]    > at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1158)
[junit4:junit4]    > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:226)
[junit4:junit4]    > at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:141)
[junit4:junit4]    > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]    > at org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:135)
[junit4:junit4]    > at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
[junit4:junit4]    > at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
[junit4:junit4]    > at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
[junit4:junit4]    > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
[junit4:junit4]    > at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:275)
[junit4:junit4]    > at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1074)
[junit4:junit4]    > at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
[junit4:junit4]    > at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
[junit4:junit4]    > at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
[junit4:junit4]    > at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
[junit4:junit4]    > at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
[junit4:junit4]    > at org.eclipse.jetty.server.Server.handle(Server.java:365)
[junit4:junit4]    > at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
[junit4:junit4]    > at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
[junit4:junit4]    > at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
[junit4:junit4]    > at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
[junit4:junit4]    > at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:948)
[junit4:junit4]    > at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
[junit4:junit4]    > at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
[junit4:junit4]    > at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
[junit4:junit4]    > at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
[junit4:junit4]    > at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]    > Caused by: java.lang.AssertionError: Directory not closed: MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/OpenCloseCoreStressTest_/00016_core/data/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@727cc402)
[junit4:junit4]    > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > at org.apache.lucene.util.CloseableDirectory.close(CloseableDirectory.java:47)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:602)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$2$1.apply(RandomizedRunner.java:599)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedContext.closeResources(RandomizedContext.java:167)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$2.afterAlways(RandomizedRunner.java:615)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > ... 1 more
[junit4:junit4] Completed on J1 in 786.41s, 6 tests, 1 error, 2 skipped <<< FAILURES!

[...truncated 667 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:388: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:877: There were test failures: 272 suites, 1144 tests, 1 suite-level error, 1 failure, 26 ignored (7 assumptions)

Total time: 147 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 208 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/208/

1 tests failed.
FAILED:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:13698/collection1lastClient and got 5 from http://127.0.0.1:13704/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:13698/collection1lastClient and got 5 from http://127.0.0.1:13704/collection1
        at __randomizedtesting.SeedInfo.seed([7ADAB6D4D7D0DD92:FB3C38CCA08FBDAE]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:235)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8813 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 1 T2671 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 7 T2671 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1363293371509
[junit4:junit4]   2> 8 T2671 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 9 T2672 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 14 T2672 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 14 T2672 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 15 T2672 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 15 T2672 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 17 T2672 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-1363293371508/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 110 T2671 oasc.ZkTestServer.run start zk server on port:13653
[junit4:junit4]   2> 111 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@eee2993
[junit4:junit4]   2> 112 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 112 T2677 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 113 T2677 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 113 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13654
[junit4:junit4]   2> 113 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13654
[junit4:junit4]   2> 114 T2675 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 118 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0000 with negotiated timeout 10000 for client /140.211.11.196:13654
[junit4:junit4]   2> 118 T2677 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0000, negotiated timeout = 10000
[junit4:junit4]   2> 119 T2678 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eee2993 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 119 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 119 T2671 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 134 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f0000
[junit4:junit4]   2> 136 T2678 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 137 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13654 which had sessionid 0x13d6a9dfc7f0000
[junit4:junit4]   2> 136 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f0000 closed
[junit4:junit4]   2> 138 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@425998ab
[junit4:junit4]   2> 138 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 139 T2679 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 139 T2679 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 140 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13655
[junit4:junit4]   2> 140 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13655
[junit4:junit4]   2> 142 T2679 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0001, negotiated timeout = 10000
[junit4:junit4]   2> 142 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0001 with negotiated timeout 10000 for client /140.211.11.196:13655
[junit4:junit4]   2> 143 T2680 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@425998ab name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 143 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 144 T2671 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 150 T2671 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 154 T2671 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 159 T2671 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 165 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 167 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 182 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 183 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 288 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 289 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 299 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 300 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 304 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 305 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 309 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 310 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 314 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 314 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 318 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 319 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 322 T2671 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 323 T2671 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 326 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f0001
[junit4:junit4]   2> 327 T2680 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 327 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13655 which had sessionid 0x13d6a9dfc7f0001
[junit4:junit4]   2> 327 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f0001 closed
[junit4:junit4]   2> 561 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 566 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13656
[junit4:junit4]   2> 567 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 567 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 568 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829
[junit4:junit4]   2> 568 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/solr.xml
[junit4:junit4]   2> 569 T2671 oasc.CoreContainer.<init> New CoreContainer 1007816338
[junit4:junit4]   2> 569 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/'
[junit4:junit4]   2> 570 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/'
[junit4:junit4]   2> 642 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 643 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 643 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 644 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 644 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 645 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 645 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 646 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 646 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 647 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 671 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 693 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 694 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 694 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@7a3f971d
[junit4:junit4]   2> 696 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 696 T2690 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 697 T2690 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 697 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13657
[junit4:junit4]   2> 698 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13657
[junit4:junit4]   2> 703 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0002 with negotiated timeout 20000 for client /140.211.11.196:13657
[junit4:junit4]   2> 703 T2690 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0002, negotiated timeout = 20000
[junit4:junit4]   2> 703 T2691 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7a3f971d name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 703 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 705 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f0002
[junit4:junit4]   2> 714 T2691 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 714 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13657 which had sessionid 0x13d6a9dfc7f0002
[junit4:junit4]   2> 714 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f0002 closed
[junit4:junit4]   2> 714 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 727 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@2cf4dafb
[junit4:junit4]   2> 728 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 729 T2692 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 729 T2692 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 730 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13658
[junit4:junit4]   2> 730 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13658
[junit4:junit4]   2> 732 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0003 with negotiated timeout 20000 for client /140.211.11.196:13658
[junit4:junit4]   2> 732 T2692 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0003, negotiated timeout = 20000
[junit4:junit4]   2> 733 T2693 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2cf4dafb name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 733 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 735 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 737 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 742 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13656_
[junit4:junit4]   2> 742 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13656_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13656_
[junit4:junit4]   2> 744 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13656_
[junit4:junit4]   2> 754 T2671 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 766 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 772 T2671 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 775 T2671 oasc.Overseer.start Overseer (id=89344794395869187-127.0.0.1:13656_-n_0000000000) starting
[junit4:junit4]   2> 776 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 778 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 786 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 788 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 789 T2695 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 790 T2671 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 804 T2671 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 807 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 811 T2694 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 814 T2696 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/collection1
[junit4:junit4]   2> 815 T2696 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 815 T2696 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 815 T2696 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 817 T2696 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/collection1/'
[junit4:junit4]   2> 818 T2696 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 819 T2696 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/collection1/lib/README' to classloader
[junit4:junit4]   2> 878 T2696 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 952 T2696 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 1056 T2696 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 1064 T2696 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1819 T2696 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1827 T2696 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1831 T2696 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1845 T2696 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1849 T2696 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1853 T2696 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1854 T2696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1854 T2696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1854 T2696 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1856 T2696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1856 T2696 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1856 T2696 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1856 T2696 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1363293371829/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/control/data/
[junit4:junit4]   2> 1856 T2696 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2658dd2d
[junit4:junit4]   2> 1857 T2696 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1857 T2696 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/control/data forceNew: false
[junit4:junit4]   2> 1857 T2696 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363293371508/control/data/index/
[junit4:junit4]   2> 1857 T2696 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363293371508/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1858 T2696 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/control/data/index forceNew: false
[junit4:junit4]   2> 1860 T2696 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3055a3d6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@42fd5260),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1860 T2696 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1862 T2696 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1862 T2696 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1862 T2696 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1863 T2696 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1863 T2696 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1863 T2696 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1863 T2696 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1864 T2696 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1864 T2696 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1869 T2696 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1873 T2696 oass.SolrIndexSearcher.<init> Opening Searcher@118cedce main
[junit4:junit4]   2> 1873 T2696 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/control/data/tlog
[junit4:junit4]   2> 1874 T2696 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1874 T2696 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1877 T2697 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@118cedce main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1878 T2696 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1878 T2696 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1879 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2316 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 2316 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13656_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13656"}
[junit4:junit4]   2> 2317 T2694 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2317 T2694 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2317 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2336 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2894 T2696 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2894 T2696 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13656 collection:control_collection shard:shard1
[junit4:junit4]   2> 2895 T2696 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2915 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2916 T2696 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2916 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2920 T2696 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2920 T2696 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2920 T2696 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13656/collection1/
[junit4:junit4]   2> 2920 T2696 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2921 T2696 oasc.SyncStrategy.syncToMe http://127.0.0.1:13656/collection1/ has no replicas
[junit4:junit4]   2> 2921 T2696 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13656/collection1/
[junit4:junit4]   2> 2921 T2696 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2927 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3842 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 3890 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3949 T2696 oasc.ZkController.register We are http://127.0.0.1:13656/collection1/ and leader is http://127.0.0.1:13656/collection1/
[junit4:junit4]   2> 3949 T2696 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13656
[junit4:junit4]   2> 3949 T2696 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3950 T2696 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3950 T2696 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3952 T2696 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 3954 T2671 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 3955 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3955 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3971 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3973 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@720116b6
[junit4:junit4]   2> 3973 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3974 T2699 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3975 T2699 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 3975 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13660
[junit4:junit4]   2> 3975 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13660
[junit4:junit4]   2> 3977 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0004 with negotiated timeout 10000 for client /140.211.11.196:13660
[junit4:junit4]   2> 3977 T2699 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0004, negotiated timeout = 10000
[junit4:junit4]   2> 3977 T2700 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@720116b6 name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3977 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3979 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 3981 T2671 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4193 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4196 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13661
[junit4:junit4]   2> 4197 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4198 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4198 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483
[junit4:junit4]   2> 4198 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/solr.xml
[junit4:junit4]   2> 4199 T2671 oasc.CoreContainer.<init> New CoreContainer 292605757
[junit4:junit4]   2> 4200 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/'
[junit4:junit4]   2> 4200 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/'
[junit4:junit4]   2> 4260 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4261 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4262 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4262 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4263 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4263 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4264 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4264 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4265 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4265 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4287 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4308 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 4309 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4310 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@97034f7
[junit4:junit4]   2> 4311 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4311 T2710 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4312 T2710 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 4313 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13662
[junit4:junit4]   2> 4313 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13662
[junit4:junit4]   2> 4315 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0005 with negotiated timeout 20000 for client /140.211.11.196:13662
[junit4:junit4]   2> 4315 T2710 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0005, negotiated timeout = 20000
[junit4:junit4]   2> 4315 T2711 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@97034f7 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4316 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4317 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f0005
[junit4:junit4]   2> 4326 T2711 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4326 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13662 which had sessionid 0x13d6a9dfc7f0005
[junit4:junit4]   2> 4326 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f0005 closed
[junit4:junit4]   2> 4327 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4336 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@58f587bb
[junit4:junit4]   2> 4337 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4338 T2712 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4338 T2712 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 4338 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13663
[junit4:junit4]   2> 4339 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13663
[junit4:junit4]   2> 4340 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0006 with negotiated timeout 20000 for client /140.211.11.196:13663
[junit4:junit4]   2> 4340 T2712 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0006, negotiated timeout = 20000
[junit4:junit4]   2> 4341 T2713 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58f587bb name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4341 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4342 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4344 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4347 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 5353 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13661_
[junit4:junit4]   2> 5359 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13661_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13661_
[junit4:junit4]   2> 5361 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13661_
[junit4:junit4]   2> 5381 T2693 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5381 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5386 T2713 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5388 T2700 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5427 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 5432 T2714 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/collection1
[junit4:junit4]   2> 5432 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13656_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13656"}
[junit4:junit4]   2> 5432 T2714 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5434 T2714 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5435 T2714 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5450 T2714 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/collection1/'
[junit4:junit4]   2> 5459 T2714 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/collection1/lib/README' to classloader
[junit4:junit4]   2> 5463 T2714 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5470 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5470 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5471 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5700 T2714 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 5811 T2714 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5912 T2714 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5920 T2714 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 6655 T2714 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 6666 T2714 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 6671 T2714 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 6687 T2714 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6693 T2714 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 6698 T2714 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6700 T2714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6701 T2714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6701 T2714 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 6702 T2714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 6703 T2714 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 6703 T2714 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6703 T2714 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1363293375483/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty1/
[junit4:junit4]   2> 6704 T2714 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2658dd2d
[junit4:junit4]   2> 6704 T2714 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 6705 T2714 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty1 forceNew: false
[junit4:junit4]   2> 6705 T2714 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty1/index/
[junit4:junit4]   2> 6705 T2714 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 6706 T2714 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty1/index forceNew: false
[junit4:junit4]   2> 6708 T2714 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6709 T2714 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6711 T2714 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6711 T2714 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6712 T2714 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6712 T2714 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6713 T2714 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6713 T2714 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6714 T2714 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6714 T2714 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6715 T2714 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6722 T2714 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6728 T2714 oass.SolrIndexSearcher.<init> Opening Searcher@452c8361 main
[junit4:junit4]   2> 6728 T2714 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty1/tlog
[junit4:junit4]   2> 6729 T2714 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6729 T2714 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6733 T2715 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@452c8361 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6735 T2714 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6735 T2714 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6987 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 6989 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13661_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13661"}
[junit4:junit4]   2> 6989 T2694 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 6989 T2694 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6996 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6996 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6996 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7739 T2714 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7739 T2714 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13661 collection:collection1 shard:shard1
[junit4:junit4]   2> 7741 T2714 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7762 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7764 T2714 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7765 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7776 T2714 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7776 T2714 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7776 T2714 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 7777 T2714 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7777 T2714 oasc.SyncStrategy.syncToMe http://127.0.0.1:13661/collection1/ has no replicas
[junit4:junit4]   2> 7777 T2714 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 7777 T2714 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7783 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8508 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8552 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8552 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8552 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8561 T2714 oasc.ZkController.register We are http://127.0.0.1:13661/collection1/ and leader is http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 8561 T2714 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13661
[junit4:junit4]   2> 8561 T2714 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8561 T2714 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8562 T2714 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8564 T2714 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8565 T2671 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 8566 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8566 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8801 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8803 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13665
[junit4:junit4]   2> 8805 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8805 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8806 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084
[junit4:junit4]   2> 8806 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/solr.xml
[junit4:junit4]   2> 8807 T2671 oasc.CoreContainer.<init> New CoreContainer 537842846
[junit4:junit4]   2> 8807 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/'
[junit4:junit4]   2> 8808 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/'
[junit4:junit4]   2> 8864 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8865 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8865 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8866 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8866 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8867 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8867 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8868 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8868 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8869 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8893 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8914 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 8915 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8916 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@2a6d76e0
[junit4:junit4]   2> 8917 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8917 T2726 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8918 T2726 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 8919 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13666
[junit4:junit4]   2> 8919 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13666
[junit4:junit4]   2> 8921 T2726 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0007, negotiated timeout = 20000
[junit4:junit4]   2> 8921 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0007 with negotiated timeout 20000 for client /140.211.11.196:13666
[junit4:junit4]   2> 8922 T2727 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2a6d76e0 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8922 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8923 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f0007
[junit4:junit4]   2> 8925 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13666 which had sessionid 0x13d6a9dfc7f0007
[junit4:junit4]   2> 8925 T2727 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8925 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f0007 closed
[junit4:junit4]   2> 8925 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8938 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@66eb63f8
[junit4:junit4]   2> 8939 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8940 T2728 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8940 T2728 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 8941 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13667
[junit4:junit4]   2> 8941 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13667
[junit4:junit4]   2> 8943 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0008 with negotiated timeout 20000 for client /140.211.11.196:13667
[junit4:junit4]   2> 8943 T2728 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0008, negotiated timeout = 20000
[junit4:junit4]   2> 8943 T2729 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@66eb63f8 name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8943 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8944 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8946 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8955 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 9959 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13665_
[junit4:junit4]   2> 9960 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13665_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13665_
[junit4:junit4]   2> 9965 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13665_
[junit4:junit4]   2> 9979 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9979 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9979 T2729 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9979 T2693 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9979 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9980 T2700 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9980 T2713 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9987 T2730 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/collection1
[junit4:junit4]   2> 9988 T2730 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9988 T2730 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9988 T2730 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9990 T2730 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/collection1/'
[junit4:junit4]   2> 9991 T2730 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/collection1/lib/README' to classloader
[junit4:junit4]   2> 9991 T2730 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 10048 T2730 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 10060 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 10061 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13661_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13661"}
[junit4:junit4]   2> 10078 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10078 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10078 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10078 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10113 T2730 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 10214 T2730 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 10220 T2730 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 10828 T2730 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10836 T2730 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10839 T2730 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10850 T2730 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10854 T2730 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10857 T2730 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10858 T2730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10859 T2730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10859 T2730 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10859 T2730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10860 T2730 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10860 T2730 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10860 T2730 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1363293380084/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2/
[junit4:junit4]   2> 10860 T2730 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2658dd2d
[junit4:junit4]   2> 10861 T2730 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10861 T2730 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2 forceNew: false
[junit4:junit4]   2> 10861 T2730 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2/index/
[junit4:junit4]   2> 10861 T2730 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10862 T2730 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2/index forceNew: false
[junit4:junit4]   2> 10863 T2730 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@653a0728 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a00792d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10864 T2730 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10865 T2730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10865 T2730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10866 T2730 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10866 T2730 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10867 T2730 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10867 T2730 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10867 T2730 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10867 T2730 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10868 T2730 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10874 T2730 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10878 T2730 oass.SolrIndexSearcher.<init> Opening Searcher@bfebbac main
[junit4:junit4]   2> 10878 T2730 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2/tlog
[junit4:junit4]   2> 10879 T2730 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10879 T2730 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10882 T2731 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@bfebbac main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10884 T2730 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10884 T2730 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11582 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 11583 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13665_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13665"}
[junit4:junit4]   2> 11583 T2694 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 11583 T2694 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 11602 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11602 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11602 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11602 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11895 T2730 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11895 T2730 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13665 collection:collection1 shard:shard1
[junit4:junit4]   2> 11899 T2730 oasc.ZkController.register We are http://127.0.0.1:13665/collection1/ and leader is http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 11899 T2730 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13665
[junit4:junit4]   2> 11900 T2730 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 11900 T2730 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C189 name=collection1 org.apache.solr.core.SolrCore@62219a0d url=http://127.0.0.1:13665/collection1 node=127.0.0.1:13665_ C189_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:13665_, base_url=http://127.0.0.1:13665}
[junit4:junit4]   2> 11900 T2732 C189 P13665 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 11901 T2730 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 11901 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 11901 T2732 C189 P13665 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 11901 T2732 C189 P13665 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11902 T2671 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 11902 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 11903 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 11910 T2732 C189 P13665 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 12119 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 12122 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13670
[junit4:junit4]   2> 12123 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 12123 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 12124 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420
[junit4:junit4]   2> 12124 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/solr.xml
[junit4:junit4]   2> 12124 T2671 oasc.CoreContainer.<init> New CoreContainer 813728562
[junit4:junit4]   2> 12125 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/'
[junit4:junit4]   2> 12125 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/'
[junit4:junit4]   2> 12176 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 12177 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 12177 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 12178 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 12178 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 12178 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 12179 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 12179 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 12180 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 12180 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 12201 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 12220 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 12220 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 12221 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@18e213a9
[junit4:junit4]   2> 12222 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12222 T2743 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12223 T2743 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 12223 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13671
[junit4:junit4]   2> 12224 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13671
[junit4:junit4]   2> 12226 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0009 with negotiated timeout 20000 for client /140.211.11.196:13671
[junit4:junit4]   2> 12226 T2743 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0009, negotiated timeout = 20000
[junit4:junit4]   2> 12227 T2744 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@18e213a9 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12227 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12228 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f0009
[junit4:junit4]   2> 12236 T2744 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 12236 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13671 which had sessionid 0x13d6a9dfc7f0009
[junit4:junit4]   2> 12236 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f0009 closed
[junit4:junit4]   2> 12237 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 12248 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@51957d2
[junit4:junit4]   2> 12249 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 12249 T2745 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 12250 T2745 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 12250 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13672
[junit4:junit4]   2> 12250 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13672
[junit4:junit4]   2> 12251 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f000a with negotiated timeout 20000 for client /140.211.11.196:13672
[junit4:junit4]   2> 12252 T2745 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f000a, negotiated timeout = 20000
[junit4:junit4]   2> 12252 T2746 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51957d2 name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 12252 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 12253 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000a type:create cxid:0x1 zxid:0x81 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12254 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000a type:create cxid:0x2 zxid:0x82 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 12257 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 13106 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 13107 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13665_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13665"}
[junit4:junit4]   2> 13123 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13123 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13260 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13670_
[junit4:junit4]   2> 13261 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000a type:delete cxid:0xd zxid:0x87 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13670_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13670_
[junit4:junit4]   2> 13262 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13670_
[junit4:junit4]   2> 13268 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13268 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13268 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13268 T2729 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13268 T2693 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13268 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13269 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 13269 T2746 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13269 T2700 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13271 T2713 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 13276 T2747 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/collection1
[junit4:junit4]   2> 13276 T2747 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 13277 T2747 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 13277 T2747 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 13278 T2747 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/collection1/'
[junit4:junit4]   2> 13279 T2747 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/collection1/lib/README' to classloader
[junit4:junit4]   2> 13280 T2747 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 13325 T2747 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 13379 T2747 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 13480 T2747 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 13486 T2747 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 13925 T2703 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=3&state=recovering&nodeName=127.0.0.1:13665_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4]   2> 14026 T2747 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 14035 T2747 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 14039 T2747 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 14052 T2747 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14059 T2747 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 14065 T2747 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14067 T2747 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 14067 T2747 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 14068 T2747 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 14069 T2747 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 14069 T2747 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 14069 T2747 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 14070 T2747 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1363293383420/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3/
[junit4:junit4]   2> 14070 T2747 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2658dd2d
[junit4:junit4]   2> 14071 T2747 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 14072 T2747 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3 forceNew: false
[junit4:junit4]   2> 14072 T2747 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3/index/
[junit4:junit4]   2> 14072 T2747 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 14073 T2747 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3/index forceNew: false
[junit4:junit4]   2> 14077 T2747 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12c6123d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e0f2b80),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 14077 T2747 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 14080 T2747 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 14081 T2747 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 14081 T2747 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 14082 T2747 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 14083 T2747 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 14083 T2747 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 14083 T2747 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 14084 T2747 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 14085 T2747 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 14102 T2747 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 14117 T2747 oass.SolrIndexSearcher.<init> Opening Searcher@1ebccff8 main
[junit4:junit4]   2> 14117 T2747 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3/tlog
[junit4:junit4]   2> 14119 T2747 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 14119 T2747 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 14124 T2748 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ebccff8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 14127 T2747 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 14127 T2747 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 14629 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 14630 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13670_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13670"}
[junit4:junit4]   2> 14630 T2694 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 14630 T2694 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 14648 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14648 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14648 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14648 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14648 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 15159 T2747 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 15159 T2747 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13670 collection:collection1 shard:shard1
[junit4:junit4]   2> 15163 T2747 oasc.ZkController.register We are http://127.0.0.1:13670/collection1/ and leader is http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 15163 T2747 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13670
[junit4:junit4]   2> 15164 T2747 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 15164 T2747 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C190 name=collection1 org.apache.solr.core.SolrCore@516e17a5 url=http://127.0.0.1:13670/collection1 node=127.0.0.1:13670_ C190_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:13670_, base_url=http://127.0.0.1:13670}
[junit4:junit4]   2> 15164 T2749 C190 P13670 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 15165 T2747 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 15165 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 15165 T2749 C190 P13670 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 15166 T2749 C190 P13670 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 15166 T2671 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 15167 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 15168 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 15173 T2749 C190 P13670 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 15495 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 15499 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13691
[junit4:junit4]   2> 15500 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 15500 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 15501 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703
[junit4:junit4]   2> 15501 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/solr.xml
[junit4:junit4]   2> 15501 T2671 oasc.CoreContainer.<init> New CoreContainer 1782731087
[junit4:junit4]   2> 15502 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/'
[junit4:junit4]   2> 15503 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/'
[junit4:junit4]   2> 15577 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 15578 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 15579 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 15579 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 15579 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 15580 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 15580 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 15581 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 15581 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 15581 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 15608 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 15631 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 15631 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 15632 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@4de9357
[junit4:junit4]   2> 15634 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15634 T2760 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 15635 T2760 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 15636 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13692
[junit4:junit4]   2> 15636 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13692
[junit4:junit4]   2> 15638 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f000b with negotiated timeout 20000 for client /140.211.11.196:13692
[junit4:junit4]   2> 15638 T2760 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f000b, negotiated timeout = 20000
[junit4:junit4]   2> 15639 T2761 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4de9357 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15639 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15640 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f000b
[junit4:junit4]   2> 15641 T2761 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 15641 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13692 which had sessionid 0x13d6a9dfc7f000b
[junit4:junit4]   2> 15641 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f000b closed
[junit4:junit4]   2> 15642 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 15653 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@3c31541b
[junit4:junit4]   2> 15654 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 15654 T2762 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 15654 T2762 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 15655 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13693
[junit4:junit4]   2> 15655 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13693
[junit4:junit4]   2> 15656 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f000c with negotiated timeout 20000 for client /140.211.11.196:13693
[junit4:junit4]   2> 15656 T2762 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f000c, negotiated timeout = 20000
[junit4:junit4]   2> 15657 T2763 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3c31541b name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 15657 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 15658 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000c type:create cxid:0x1 zxid:0x94 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15659 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000c type:create cxid:0x2 zxid:0x95 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 15668 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2>  C189_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:13665_, base_url=http://127.0.0.1:13665}
[junit4:junit4]   2> 15926 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:13661/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 15926 T2732 C189 P13665 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:13665 START replicas=[http://127.0.0.1:13661/collection1/] nUpdates=100
[junit4:junit4]   2> 15926 T2732 C189 P13665 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 15927 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 15927 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 15927 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 15928 T2732 C189 P13665 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 15928 T2732 C189 P13665 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:13661/collection1/. core=collection1
[junit4:junit4]   2> 15928 T2732 C189 P13665 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C191 name=collection1 org.apache.solr.core.SolrCore@278ece77 url=http://127.0.0.1:13661/collection1 node=127.0.0.1:13661_ C191_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:13661_, base_url=http://127.0.0.1:13661, leader=true}
[junit4:junit4]   2> 15935 T2705 C191 P13661 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 15941 T2706 C191 P13661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 15944 T2706 C191 P13661 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15945 T2706 C191 P13661 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15946 T2706 C191 P13661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 15946 T2706 C191 P13661 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 15947 T2706 C191 P13661 oass.SolrIndexSearcher.<init> Opening Searcher@6b4fe04a realtime
[junit4:junit4]   2> 15947 T2706 C191 P13661 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 15948 T2706 C191 P13661 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 15950 T2732 C189 P13665 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15950 T2732 C189 P13665 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 15952 T2707 C191 P13661 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 15953 T2707 C191 P13661 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4]   2> 15953 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 15953 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 15953 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 15956 T2707 C191 P13661 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4]   2> 15957 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 15958 T2732 C189 P13665 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty2/index.20130314143627458 forceNew: false
[junit4:junit4]   2> 15958 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ca1760a lockFactory=org.apache.lucene.store.NativeFSLockFactory@f2ed366) fullCopy=false
[junit4:junit4]   2> 15960 T2707 C191 P13661 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4]   2> 15961 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 15971 T2732 C189 P13665 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 15971 T2732 C189 P13665 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 16024 T2732 C189 P13665 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@653a0728 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a00792d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@653a0728 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a00792d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 16025 T2732 C189 P13665 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 16025 T2732 C189 P13665 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 16025 T2732 C189 P13665 oass.SolrIndexSearcher.<init> Opening Searcher@3dbd423e main
[junit4:junit4]   2> 16026 T2731 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3dbd423e main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 16027 T2732 C189 P13665 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@4ca1760a lockFactory=org.apache.lucene.store.NativeFSLockFactory@f2ed366)
[junit4:junit4]   2> 16027 T2732 C189 P13665 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 16027 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 16027 T2732 C189 P13665 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 16028 T2732 C189 P13665 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16062 T2732 C189 P13665 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 16159 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 16159 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13670_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13670"}
[junit4:junit4]   2> 16201 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13665_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13665"}
[junit4:junit4]   2> 16289 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16289 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16289 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16289 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16289 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16289 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16672 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13691_
[junit4:junit4]   2> 16673 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000c type:delete cxid:0xd zxid:0x9e txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13691_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13691_
[junit4:junit4]   2> 16675 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13691_
[junit4:junit4]   2> 16677 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16677 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16677 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16678 T2763 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16678 T2693 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16678 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16678 T2729 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16678 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16678 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 16679 T2713 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16680 T2746 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16685 T2700 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 16691 T2766 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/collection1
[junit4:junit4]   2> 16691 T2766 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 16692 T2766 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 16692 T2766 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 16693 T2766 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/collection1/'
[junit4:junit4]   2> 16695 T2766 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/collection1/lib/README' to classloader
[junit4:junit4]   2> 16696 T2766 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 16750 T2766 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 16816 T2766 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 16917 T2766 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 16925 T2766 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 17206 T2704 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=4&state=recovering&nodeName=127.0.0.1:13670_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4]   2> 17660 T2766 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 17677 T2766 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 17682 T2766 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 17696 T2766 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 17701 T2766 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 17707 T2766 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 17708 T2766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 17708 T2766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 17709 T2766 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 17710 T2766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 17710 T2766 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 17710 T2766 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 17711 T2766 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1363293386703/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4/
[junit4:junit4]   2> 17711 T2766 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2658dd2d
[junit4:junit4]   2> 17711 T2766 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 17712 T2766 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4 forceNew: false
[junit4:junit4]   2> 17712 T2766 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4/index/
[junit4:junit4]   2> 17712 T2766 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 17713 T2766 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4/index forceNew: false
[junit4:junit4]   2> 17716 T2766 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@21917d40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4841c79a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 17717 T2766 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 17719 T2766 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 17719 T2766 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 17720 T2766 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 17721 T2766 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 17721 T2766 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 17721 T2766 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 17722 T2766 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 17722 T2766 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 17722 T2766 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 17733 T2766 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 17741 T2766 oass.SolrIndexSearcher.<init> Opening Searcher@417d0498 main
[junit4:junit4]   2> 17741 T2766 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4/tlog
[junit4:junit4]   2> 17742 T2766 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 17742 T2766 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 17746 T2767 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@417d0498 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 17748 T2766 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 17749 T2766 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 17801 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 17801 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13691_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13691"}
[junit4:junit4]   2> 17802 T2694 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 17802 T2694 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 17826 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17827 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17827 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17827 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17827 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17827 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 18752 T2766 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 18752 T2766 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13691 collection:collection1 shard:shard1
[junit4:junit4]   2> 18756 T2766 oasc.ZkController.register We are http://127.0.0.1:13691/collection1/ and leader is http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 18757 T2766 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13691
[junit4:junit4]   2> 18757 T2766 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 18757 T2766 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C192 name=collection1 org.apache.solr.core.SolrCore@6087febc url=http://127.0.0.1:13691/collection1 node=127.0.0.1:13691_ C192_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:13691_, base_url=http://127.0.0.1:13691}
[junit4:junit4]   2> 18757 T2768 C192 P13691 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 18758 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 18758 T2766 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 18758 T2768 C192 P13691 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 18759 T2768 C192 P13691 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 18759 T2671 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 18760 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 18760 T2768 C192 P13691 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 18761 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19041 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 19044 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13698
[junit4:junit4]   2> 19045 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 19045 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 19046 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284
[junit4:junit4]   2> 19046 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/solr.xml
[junit4:junit4]   2> 19047 T2671 oasc.CoreContainer.<init> New CoreContainer 1972326028
[junit4:junit4]   2> 19047 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/'
[junit4:junit4]   2> 19048 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/'
[junit4:junit4]   2> 19106 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 19106 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 19107 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 19107 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 19108 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 19108 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 19109 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 19109 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 19109 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 19110 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 19130 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 19150 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 19151 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 19151 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@1c335557
[junit4:junit4]   2> 19152 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 19153 T2779 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 19154 T2779 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 19154 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13699
[junit4:junit4]   2> 19154 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13699
[junit4:junit4]   2> 19155 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f000d with negotiated timeout 20000 for client /140.211.11.196:13699
[junit4:junit4]   2> 19155 T2779 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f000d, negotiated timeout = 20000
[junit4:junit4]   2> 19156 T2780 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1c335557 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 19156 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 19157 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f000d
[junit4:junit4]   2> 19158 T2780 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 19158 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13699 which had sessionid 0x13d6a9dfc7f000d
[junit4:junit4]   2> 19158 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f000d closed
[junit4:junit4]   2> 19158 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 19169 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@1cd75ff9
[junit4:junit4]   2> 19170 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 19170 T2781 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 19171 T2781 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 19171 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13700
[junit4:junit4]   2> 19172 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13700
[junit4:junit4]   2> 19174 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f000e with negotiated timeout 20000 for client /140.211.11.196:13700
[junit4:junit4]   2> 19174 T2781 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f000e, negotiated timeout = 20000
[junit4:junit4]   2> 19174 T2782 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1cd75ff9 name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 19174 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 19175 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000e type:create cxid:0x1 zxid:0xab txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 19182 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000e type:create cxid:0x2 zxid:0xac txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 19189 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2>  C190_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:13670_, base_url=http://127.0.0.1:13670}
[junit4:junit4]   2> 19208 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:13661/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 19208 T2749 C190 P13670 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:13670 START replicas=[http://127.0.0.1:13661/collection1/] nUpdates=100
[junit4:junit4]   2> 19208 T2749 C190 P13670 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 19209 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 19209 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 19209 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 19209 T2749 C190 P13670 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 19209 T2749 C190 P13670 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:13661/collection1/. core=collection1
[junit4:junit4]   2> 19209 T2749 C190 P13670 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19211 T2705 C191 P13661 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 19221 T2709 C191 P13661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 19224 T2709 C191 P13661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 19224 T2709 C191 P13661 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 19225 T2709 C191 P13661 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 19225 T2709 C191 P13661 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 19226 T2749 C190 P13670 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 19226 T2749 C190 P13670 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 19228 T2707 C191 P13661 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 19228 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex Master's generation: 3
[junit4:junit4]   2> 19229 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 19229 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 19231 T2707 C191 P13661 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=3&version=2} status=0 QTime=0
[junit4:junit4]   2> 19231 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 19232 T2749 C190 P13670 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty3/index.20130314143630733 forceNew: false
[junit4:junit4]   2> 19232 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@655f676b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d0630d7) fullCopy=false
[junit4:junit4]   2> 19235 T2707 C191 P13661 REQ /replication {file=segments_3&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=3} status=0 QTime=0
[junit4:junit4]   2> 19236 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 19246 T2749 C190 P13670 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 19246 T2749 C190 P13670 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 19299 T2749 C190 P13670 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12c6123d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e0f2b80),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@12c6123d lockFactory=org.apache.lucene.store.NativeFSLockFactory@1e0f2b80),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 19300 T2749 C190 P13670 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 19300 T2749 C190 P13670 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 19300 T2749 C190 P13670 oass.SolrIndexSearcher.<init> Opening Searcher@240f2262 main
[junit4:junit4]   2> 19302 T2748 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@240f2262 main{StandardDirectoryReader(segments_3:1:nrt)}
[junit4:junit4]   2> 19302 T2749 C190 P13670 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@655f676b lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d0630d7)
[junit4:junit4]   2> 19303 T2749 C190 P13670 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 19303 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 19303 T2749 C190 P13670 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 19303 T2749 C190 P13670 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19309 T2749 C190 P13670 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 19337 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 19338 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13691_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13691"}
[junit4:junit4]   2> 19355 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13670_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13670"}
[junit4:junit4]   2> 19369 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19369 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19369 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19369 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19369 T2782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19369 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19369 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19784 T2708 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=5&state=recovering&nodeName=127.0.0.1:13691_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 20192 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13698_
[junit4:junit4]   2> 20194 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f000e type:delete cxid:0xd zxid:0xb5 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13698_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13698_
[junit4:junit4]   2> 20195 T2671 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:13698_
[junit4:junit4]   2> 20205 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20205 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20205 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20206 T2782 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20206 T2763 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20206 T2693 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20206 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 20206 T2782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 20206 T2729 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20206 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 20207 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 20208 T2746 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20208 T2700 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20208 T2713 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 20215 T2785 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/collection1
[junit4:junit4]   2> 20216 T2785 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 20216 T2785 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 20216 T2785 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 20218 T2785 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/collection1/'
[junit4:junit4]   2> 20220 T2785 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/collection1/lib/README' to classloader
[junit4:junit4]   2> 20221 T2785 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 20285 T2785 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 20357 T2785 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 20458 T2785 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 20464 T2785 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 21205 T2785 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 21219 T2785 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 21224 T2785 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 21238 T2785 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 21244 T2785 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 21249 T2785 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 21251 T2785 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 21251 T2785 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 21252 T2785 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 21253 T2785 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 21253 T2785 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 21253 T2785 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 21254 T2785 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty5-1363293390284/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty5/
[junit4:junit4]   2> 21254 T2785 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@2658dd2d
[junit4:junit4]   2> 21254 T2785 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 21255 T2785 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty5 forceNew: false
[junit4:junit4]   2> 21255 T2785 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty5/index/
[junit4:junit4]   2> 21255 T2785 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 21256 T2785 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty5/index forceNew: false
[junit4:junit4]   2> 21259 T2785 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2f26ac7c lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c5838aa),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 21259 T2785 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 21261 T2785 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 21262 T2785 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 21262 T2785 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 21263 T2785 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 21264 T2785 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 21264 T2785 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 21264 T2785 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 21265 T2785 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 21265 T2785 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 21274 T2785 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 21280 T2785 oass.SolrIndexSearcher.<init> Opening Searcher@74e654ce main
[junit4:junit4]   2> 21281 T2785 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty5/tlog
[junit4:junit4]   2> 21282 T2785 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 21282 T2785 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 21287 T2786 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@74e654ce main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 21290 T2785 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 21290 T2785 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2>  C192_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:13691_, base_url=http://127.0.0.1:13691}
[junit4:junit4]   2> 21785 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:13661/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 21785 T2768 C192 P13691 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:13691 START replicas=[http://127.0.0.1:13661/collection1/] nUpdates=100
[junit4:junit4]   2> 21786 T2768 C192 P13691 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 21786 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 21786 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 21786 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 21786 T2768 C192 P13691 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 21786 T2768 C192 P13691 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:13661/collection1/. core=collection1
[junit4:junit4]   2> 21787 T2768 C192 P13691 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 21788 T2705 C191 P13661 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 21805 T2703 C191 P13661 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 21806 T2703 C191 P13661 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=3
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@61f6247e lockFactory=org.apache.lucene.store.NativeFSLockFactory@6b339fce),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 21807 T2703 C191 P13661 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 21807 T2703 C191 P13661 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 21807 T2703 C191 P13661 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> 21808 T2768 C192 P13691 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 21808 T2768 C192 P13691 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 21811 T2707 C191 P13661 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 21811 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex Master's generation: 4
[junit4:junit4]   2> 21812 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 21812 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 21814 T2707 C191 P13661 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=4&version=2} status=0 QTime=0
[junit4:junit4]   2> 21814 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 21815 T2768 C192 P13691 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1363293371508/jetty4/index.20130314143633316 forceNew: false
[junit4:junit4]   2> 21816 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(org.apache.lucene.store.RAMDirectory@3a825091 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bd6e963) fullCopy=false
[junit4:junit4]   2> 21818 T2707 C191 P13661 REQ /replication {file=segments_4&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=4} status=0 QTime=0
[junit4:junit4]   2> 21819 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 21829 T2768 C192 P13691 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 21829 T2768 C192 P13691 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 21881 T2768 C192 P13691 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@21917d40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4841c79a),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@21917d40 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4841c79a),segFN=segments_4,generation=4,filenames=[segments_4]
[junit4:junit4]   2> 21882 T2768 C192 P13691 oasc.SolrDeletionPolicy.updateCommits newest commit = 4[segments_4]
[junit4:junit4]   2> 21882 T2768 C192 P13691 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 21882 T2768 C192 P13691 oass.SolrIndexSearcher.<init> Opening Searcher@4f8e835c main
[junit4:junit4]   2> 21883 T2767 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4f8e835c main{StandardDirectoryReader(segments_4:1:nrt)}
[junit4:junit4]   2> 21884 T2768 C192 P13691 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(org.apache.lucene.store.RAMDirectory@3a825091 lockFactory=org.apache.lucene.store.NativeFSLockFactory@bd6e963)
[junit4:junit4]   2> 21884 T2768 C192 P13691 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 21884 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 21884 T2768 C192 P13691 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 21884 T2768 C192 P13691 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 21886 T2768 C192 P13691 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 22377 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 22378 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"6",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13698_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13698"}
[junit4:junit4]   2> 22379 T2694 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 22379 T2694 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 22392 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13691_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13691"}
[junit4:junit4]   2> 22412 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 22412 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 22412 T2782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 22412 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 22412 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 22412 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 22412 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23298 T2785 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 23298 T2785 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:13698 collection:collection1 shard:shard1
[junit4:junit4]   2> 23302 T2785 oasc.ZkController.register We are http://127.0.0.1:13698/collection1/ and leader is http://127.0.0.1:13661/collection1/
[junit4:junit4]   2> 23302 T2785 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:13698
[junit4:junit4]   2> 23302 T2785 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 23302 T2785 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C193 name=collection1 org.apache.solr.core.SolrCore@7fea7578 url=http://127.0.0.1:13698/collection1 node=127.0.0.1:13698_ C193_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:13698_, base_url=http://127.0.0.1:13698}
[junit4:junit4]   2> 23303 T2789 C193 P13698 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 23303 T2789 C193 P13698 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 23303 T2789 C193 P13698 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 23303 T2789 C193 P13698 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23304 T2785 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23305 T2789 C193 P13698 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 23305 T2671 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0
[junit4:junit4]   2> 23306 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 23306 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 23578 T2671 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 23581 T2671 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:13704
[junit4:junit4]   2> 23581 T2671 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 23582 T2671 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 23582 T2671 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363293394819
[junit4:junit4]   2> 23583 T2671 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363293394819/solr.xml
[junit4:junit4]   2> 23583 T2671 oasc.CoreContainer.<init> New CoreContainer 9547283
[junit4:junit4]   2> 23584 T2671 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363293394819/'
[junit4:junit4]   2> 23584 T2671 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty6-1363293394819/'
[junit4:junit4]   2> 23657 T2671 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 23658 T2671 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 23658 T2671 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 23659 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 23659 T2671 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 23660 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 23660 T2671 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 23661 T2671 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 23661 T2671 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 23661 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 23685 T2671 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 23708 T2671 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:13653/solr
[junit4:junit4]   2> 23709 T2671 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 23709 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3e8a5777
[junit4:junit4]   2> 23710 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 23711 T2800 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 23712 T2800 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 23712 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13705
[junit4:junit4]   2> 23712 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13705
[junit4:junit4]   2> 23714 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f000f with negotiated timeout 20000 for client /140.211.11.196:13705
[junit4:junit4]   2> 23714 T2800 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f000f, negotiated timeout = 20000
[junit4:junit4]   2> 23714 T2801 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3e8a5777 name:ZooKeeperConnection Watcher:127.0.0.1:13653 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 23714 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 23716 T2676 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6a9dfc7f000f
[junit4:junit4]   2> 23717 T2801 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 23717 T2673 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:13705 which had sessionid 0x13d6a9dfc7f000f
[junit4:junit4]   2> 23717 T2671 oaz.ZooKeeper.close Session: 0x13d6a9dfc7f000f closed
[junit4:junit4]   2> 23718 T2671 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 23728 T2671 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:13653/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6eb481a2
[junit4:junit4]   2> 23730 T2671 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 23730 T2802 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:13653. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 23731 T2802 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:13653, initiating session
[junit4:junit4]   2> 23731 T2673 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:13706
[junit4:junit4]   2> 23731 T2673 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:13706
[junit4:junit4]   2> 23733 T2675 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6a9dfc7f0010 with negotiated timeout 20000 for client /140.211.11.196:13706
[junit4:junit4]   2> 23733 T2802 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:13653, sessionid = 0x13d6a9dfc7f0010, negotiated timeout = 20000
[junit4:junit4]   2> 23733 T2803 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6eb481a2 name:ZooKeeperConnection Watcher:127.0.0.1:13653/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 23733 T2671 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 23734 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0010 type:create cxid:0x1 zxid:0xc6 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 23736 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0010 type:create cxid:0x2 zxid:0xc7 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 23744 T2671 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 23932 T2694 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23933 T2694 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"6",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:13698_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:13698"}
[junit4:junit4]   2> 23948 T2713 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2782 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2803 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2700 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2729 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2746 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2763 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 23948 T2693 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 24318 T2706 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=6&state=recovering&nodeName=127.0.0.1:13698_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1000
[junit4:junit4]   2> 24747 T2671 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:13704_
[junit4:junit4]   2> 24748 T2676 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6a9dfc7f0010 type:delete cxid:0xd zxid:0xcc txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:13704_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:13

[...truncated too long message...]

ver.shutdown shutting down
[junit4:junit4]   2> 641176 T2671 oazs.SessionTrackerImpl.shutdown Shutting down
[junit4:junit4]   2> 641176 T2671 oazs.PrepRequestProcessor.shutdown Shutting down
[junit4:junit4]   2> 641177 T2671 oazs.SyncRequestProcessor.shutdown Shutting down
[junit4:junit4]   2> 641177 T2671 oazs.FinalRequestProcessor.shutdown shutdown of request processor complete
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> NOTE: download the large Jenkins line-docs file by running 'ant get-jenkins-line-docs' in the lucene directory.
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=7ADAB6D4D7D0DD92 -Dtests.multiplier=2 -Dtests.nightly=true -Dtests.slow=true -Dtests.linedocsfile=/home/hudson/lucene-data/enwiki.random.lines.txt -Dtests.locale=en_GB -Dtests.timezone=America/Costa_Rica -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE  641s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:13698/collection1lastClient and got 5 from http://127.0.0.1:13704/collection1
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([7ADAB6D4D7D0DD92:FB3C38CCA08FBDAE]:0)
[junit4:junit4]    > at org.junit.Assert.fail(Assert.java:93)
[junit4:junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:963)
[junit4:junit4]    > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:235)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
[junit4:junit4]    > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[junit4:junit4]    > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[junit4:junit4]    > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[junit4:junit4]    > at java.lang.reflect.Method.invoke(Method.java:616)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
[junit4:junit4]    > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
[junit4:junit4]    > at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
[junit4:junit4]    > at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
[junit4:junit4]    > at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:679)
[junit4:junit4]   2> 641217 T2671 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 641222 T2670 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 1 leaked thread(s).
[junit4:junit4]   2> 641500 T2674 oazs.SessionTrackerImpl.run SessionTrackerImpl exited loop!
[junit4:junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=true,coord=crazy): {}, locale=en_GB, timezone=America/Costa_Rica
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=1,free=230976968,total=401670144
[junit4:junit4]   2> NOTE: All tests run in this JVM: [SOLR749Test, SoftAutoCommitTest, SimpleFacetsTest, IndexSchemaTest, TestFieldTypeCollectionResource, TestStressRecovery, BinaryUpdateRequestHandlerTest, TestSearchPerf, TestSolrIndexConfig, AlternateDirectoryTest, ClusterStateUpdateTest, UnloadDistributedZkTest, JsonLoaderTest, TestLRUCache, TestLFUCache, BasicDistributedZkTest, PolyFieldTest, TestGroupingSearch, TestPHPSerializedResponseWriter, TermsComponentTest, ZkCLITest, TestPropInjectDefaults, ReturnFieldsTest, SuggesterFSTTest, ZkControllerTest, TestRandomDVFaceting, DocValuesTest, SolrIndexSplitterTest, StatelessScriptUpdateProcessorFactoryTest, MBeansHandlerTest, SpellingQueryConverterTest, ExternalFileFieldSortTest, DistributedQueryElevationComponentTest, LegacyHTMLStripCharFilterTest, LoggingHandlerTest, TestSolrDeletionPolicy1, QueryParsingTest, LeaderElectionTest, TestSolrDeletionPolicy2, MoreLikeThisHandlerTest, TestTrie, TestStressVersions, PreAnalyzedFieldTest, ResourceLoaderTest, FileBasedSpellCheckerTest, LukeRequestHandlerTest, SpatialFilterTest, PeerSyncTest, FieldMutatingUpdateProcessorTest, BadComponentTest, TestPhraseSuggestions, TestCollationKeyRangeQueries, SynonymTokenizerTest, SystemInfoHandlerTest, SuggesterTest, TestPluginEnable, QueryElevationComponentTest, SearchHandlerTest, TestFastWriter, BasicFunctionalityTest, ShowFileRequestHandlerTest, TestFunctionQuery, TestSolrCoreProperties, TermVectorComponentTest, TestReplicationHandler, TestJoin, CollectionsAPIDistributedZkTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 641.76s, 1 test, 1 failure <<< FAILURES!

[...truncated 382 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:388: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:877: There were test failures: 273 suites, 1147 tests, 1 failure, 25 ignored (6 assumptions)

Total time: 95 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 209 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/209/

2 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
        at __randomizedtesting.SeedInfo.seed([9D3831B80E21DF17:1CDEBFA0797EBF2B]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
        at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
        at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)


FAILED:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:


Stack Trace:
java.lang.AssertionError
        at __randomizedtesting.SeedInfo.seed([9D3831B80E21DF17:1CDEBFA0797EBF2B]:0)
        at org.junit.Assert.fail(Assert.java:92)
        at org.junit.Assert.assertTrue(Assert.java:43)
        at org.junit.Assert.assertTrue(Assert.java:54)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:196)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8764 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T1078 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 5 T1078 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628
[junit4:junit4]   2> 6 T1078 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 7 T1079 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 10 T1079 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 10 T1079 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 10 T1079 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 11 T1079 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 11 T1079 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 108 T1078 oasc.ZkTestServer.run start zk server on port:56885
[junit4:junit4]   2> 111 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@42b6d03
[junit4:junit4]   2> 113 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 113 T1084 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 114 T1084 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 114 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:43946
[junit4:junit4]   2> 114 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:43946
[junit4:junit4]   2> 115 T1082 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 120 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20000 with negotiated timeout 10000 for client /140.211.11.196:43946
[junit4:junit4]   2> 120 T1084 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20000, negotiated timeout = 10000
[junit4:junit4]   2> 120 T1085 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42b6d03 name:ZooKeeperConnection Watcher:127.0.0.1:56885 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 120 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 121 T1078 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 158 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee20000
[junit4:junit4]   2> 184 T1085 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 184 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee20000 closed
[junit4:junit4]   2> 184 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:43946 which had sessionid 0x13d6f8dfee20000
[junit4:junit4]   2> 184 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@639d3030
[junit4:junit4]   2> 186 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 186 T1086 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 187 T1086 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 187 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:50327
[junit4:junit4]   2> 187 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:50327
[junit4:junit4]   2> 189 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20001 with negotiated timeout 10000 for client /140.211.11.196:50327
[junit4:junit4]   2> 189 T1086 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20001, negotiated timeout = 10000
[junit4:junit4]   2> 189 T1087 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@639d3030 name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 189 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 190 T1078 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 200 T1078 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 204 T1078 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 207 T1078 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 211 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 212 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 227 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 228 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 233 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 234 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 237 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 238 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 242 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 242 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 250 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 251 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 255 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 255 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 259 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 260 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 263 T1078 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 264 T1078 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 268 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee20001
[junit4:junit4]   2> 269 T1087 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 269 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:50327 which had sessionid 0x13d6f8dfee20001
[junit4:junit4]   2> 269 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee20001 closed
[junit4:junit4]   2> 461 T1078 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 467 T1078 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:14586
[junit4:junit4]   2> 467 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 468 T1078 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 468 T1078 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892
[junit4:junit4]   2> 469 T1078 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/solr.xml
[junit4:junit4]   2> 469 T1078 oasc.CoreContainer.<init> New CoreContainer 446463486
[junit4:junit4]   2> 470 T1078 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/'
[junit4:junit4]   2> 470 T1078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/'
[junit4:junit4]   2> 520 T1078 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 520 T1078 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 521 T1078 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 521 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 521 T1078 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 522 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 522 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 523 T1078 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 523 T1078 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 524 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 544 T1078 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 562 T1078 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:56885/solr
[junit4:junit4]   2> 562 T1078 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 563 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@37b241c1
[junit4:junit4]   2> 564 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 564 T1097 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 565 T1097 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 565 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:61231
[junit4:junit4]   2> 565 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:61231
[junit4:junit4]   2> 567 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20002 with negotiated timeout 20000 for client /140.211.11.196:61231
[junit4:junit4]   2> 567 T1097 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20002, negotiated timeout = 20000
[junit4:junit4]   2> 568 T1098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@37b241c1 name:ZooKeeperConnection Watcher:127.0.0.1:56885 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 568 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 569 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee20002
[junit4:junit4]   2> 579 T1098 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 579 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:61231 which had sessionid 0x13d6f8dfee20002
[junit4:junit4]   2> 579 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee20002 closed
[junit4:junit4]   2> 580 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 591 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@6cf71784
[junit4:junit4]   2> 592 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 592 T1099 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 593 T1099 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 593 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:36950
[junit4:junit4]   2> 593 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:36950
[junit4:junit4]   2> 594 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20003 with negotiated timeout 20000 for client /140.211.11.196:36950
[junit4:junit4]   2> 594 T1099 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20003, negotiated timeout = 20000
[junit4:junit4]   2> 595 T1100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6cf71784 name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 595 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 602 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x2 zxid:0x1a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 604 T1078 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 606 T1078 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:14586_
[junit4:junit4]   2> 607 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:delete cxid:0x8 zxid:0x1c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:14586_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:14586_
[junit4:junit4]   2> 608 T1078 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:14586_
[junit4:junit4]   2> 619 T1078 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 631 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:delete cxid:0x16 zxid:0x21 txntype:-1 reqpath:n/a Error Path:/solr/overseer_elect/leader Error:KeeperErrorCode = NoNode for /solr/overseer_elect/leader
[junit4:junit4]   2> 632 T1078 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 641 T1078 oasc.Overseer.start Overseer (id=89350223274573827-127.0.0.1:14586_-n_0000000000) starting
[junit4:junit4]   2> 642 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x1b zxid:0x23 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 643 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x1c zxid:0x24 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 649 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x1d zxid:0x25 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 650 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x1e zxid:0x26 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 658 T1102 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 659 T1078 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 663 T1078 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 674 T1078 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 677 T1101 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 681 T1103 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/collection1
[junit4:junit4]   2> 681 T1103 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 682 T1103 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 682 T1103 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 684 T1103 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/collection1/'
[junit4:junit4]   2> 685 T1103 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/collection1/lib/README' to classloader
[junit4:junit4]   2> 685 T1103 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 732 T1103 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 787 T1103 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 788 T1103 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 795 T1103 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 1398 T1103 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 1410 T1103 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 1414 T1103 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 1432 T1103 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1437 T1103 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 1442 T1103 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1443 T1103 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1443 T1103 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1444 T1103 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 1445 T1103 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 1445 T1103 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 1445 T1103 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1446 T1103 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1363376209892/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/control/data/
[junit4:junit4]   2> 1446 T1103 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11d13272
[junit4:junit4]   2> 1446 T1103 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 1447 T1103 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/control/data forceNew: false
[junit4:junit4]   2> 1447 T1103 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/control/data/index/
[junit4:junit4]   2> 1447 T1103 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 1449 T1103 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/control/data/index forceNew: false
[junit4:junit4]   2> 1454 T1103 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050053tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bdf228e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 1455 T1103 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 1457 T1103 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 1457 T1103 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 1458 T1103 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 1458 T1103 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 1459 T1103 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 1459 T1103 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 1459 T1103 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 1460 T1103 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 1460 T1103 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 1467 T1103 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 1474 T1103 oass.SolrIndexSearcher.<init> Opening Searcher@6af84412 main
[junit4:junit4]   2> 1474 T1103 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/control/data/tlog
[junit4:junit4]   2> 1475 T1103 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 1476 T1103 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 1479 T1104 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6af84412 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 1480 T1103 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 1480 T1103 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 1481 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x55 zxid:0x2a txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue Error:KeeperErrorCode = NoNode for /solr/overseer/queue
[junit4:junit4]   2> 2183 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 2184 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:14586_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:14586"}
[junit4:junit4]   2> 2184 T1101 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 2189 T1101 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 2189 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x5d zxid:0x2d txntype:-1 reqpath:n/a Error Path:/solr/overseer/queue-work Error:KeeperErrorCode = NoNode for /solr/overseer/queue-work
[junit4:junit4]   2> 2232 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 2493 T1103 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 2493 T1103 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:14586 collection:control_collection shard:shard1
[junit4:junit4]   2> 2494 T1103 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 2511 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:delete cxid:0x77 zxid:0x37 txntype:-1 reqpath:n/a Error Path:/solr/collections/control_collection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/control_collection/leaders
[junit4:junit4]   2> 2512 T1103 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 2512 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x78 zxid:0x38 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 2522 T1103 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 2522 T1103 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 2523 T1103 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:14586/collection1/
[junit4:junit4]   2> 2523 T1103 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 2523 T1103 oasc.SyncStrategy.syncToMe http://127.0.0.1:14586/collection1/ has no replicas
[junit4:junit4]   2> 2523 T1103 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:14586/collection1/
[junit4:junit4]   2> 2524 T1103 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 2528 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20003 type:create cxid:0x82 zxid:0x3c txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 3745 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 3762 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 3806 T1103 oasc.ZkController.register We are http://127.0.0.1:14586/collection1/ and leader is http://127.0.0.1:14586/collection1/
[junit4:junit4]   2> 3806 T1103 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:14586
[junit4:junit4]   2> 3806 T1103 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 3806 T1103 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 3806 T1103 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 3808 T1103 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 3810 T1078 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 3810 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 3811 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 3826 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 3832 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@8aaa801
[junit4:junit4]   2> 3833 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 3834 T1106 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 3834 T1106 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 3834 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:31749
[junit4:junit4]   2> 3835 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:31749
[junit4:junit4]   2> 3836 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20004 with negotiated timeout 10000 for client /140.211.11.196:31749
[junit4:junit4]   2> 3836 T1106 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20004, negotiated timeout = 10000
[junit4:junit4]   2> 3836 T1107 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8aaa801 name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 3836 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 3839 T1078 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 3843 T1078 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 4034 T1078 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 4037 T1078 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:36273
[junit4:junit4]   2> 4037 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 4038 T1078 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 4038 T1078 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466
[junit4:junit4]   2> 4039 T1078 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/solr.xml
[junit4:junit4]   2> 4039 T1078 oasc.CoreContainer.<init> New CoreContainer 669175310
[junit4:junit4]   2> 4040 T1078 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/'
[junit4:junit4]   2> 4040 T1078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/'
[junit4:junit4]   2> 4090 T1078 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 4090 T1078 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 4091 T1078 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 4091 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 4091 T1078 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 4092 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 4092 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 4093 T1078 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 4093 T1078 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 4094 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 4114 T1078 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 4132 T1078 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:56885/solr
[junit4:junit4]   2> 4132 T1078 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 4133 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@5bd76887
[junit4:junit4]   2> 4134 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4135 T1117 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4135 T1117 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 4136 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:17139
[junit4:junit4]   2> 4136 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:17139
[junit4:junit4]   2> 4138 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20005 with negotiated timeout 20000 for client /140.211.11.196:17139
[junit4:junit4]   2> 4138 T1117 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20005, negotiated timeout = 20000
[junit4:junit4]   2> 4138 T1118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bd76887 name:ZooKeeperConnection Watcher:127.0.0.1:56885 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4139 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4140 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee20005
[junit4:junit4]   2> 4152 T1118 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 4152 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:17139 which had sessionid 0x13d6f8dfee20005
[junit4:junit4]   2> 4152 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee20005 closed
[junit4:junit4]   2> 4153 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 4164 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@12b69af7
[junit4:junit4]   2> 4165 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 4165 T1119 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 4166 T1119 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 4166 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:35282
[junit4:junit4]   2> 4166 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:35282
[junit4:junit4]   2> 4174 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20006 with negotiated timeout 20000 for client /140.211.11.196:35282
[junit4:junit4]   2> 4175 T1119 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20006, negotiated timeout = 20000
[junit4:junit4]   2> 4175 T1120 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12b69af7 name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 4175 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 4177 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:create cxid:0x1 zxid:0x4a txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4183 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:create cxid:0x2 zxid:0x4b txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 4185 T1078 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 5189 T1078 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36273_
[junit4:junit4]   2> 5191 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:delete cxid:0xb zxid:0x4c txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:36273_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:36273_
[junit4:junit4]   2> 5196 T1078 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36273_
[junit4:junit4]   2> 5199 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 5200 T1107 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5200 T1120 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5202 T1100 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 5212 T1121 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/collection1
[junit4:junit4]   2> 5212 T1121 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 5213 T1121 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 5213 T1121 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 5215 T1121 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/collection1/'
[junit4:junit4]   2> 5216 T1121 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/collection1/lib/README' to classloader
[junit4:junit4]   2> 5216 T1121 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 5263 T1121 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 5277 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 5278 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:14586_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:14586"}
[junit4:junit4]   2> 5287 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5287 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5287 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 5320 T1121 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 5321 T1121 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 5328 T1121 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 5934 T1121 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 5946 T1121 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 5951 T1121 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 5973 T1121 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5978 T1121 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 5983 T1121 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5984 T1121 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5984 T1121 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5985 T1121 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 5986 T1121 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 5986 T1121 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 5986 T1121 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5987 T1121 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty1/
[junit4:junit4]   2> 5987 T1121 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11d13272
[junit4:junit4]   2> 5987 T1121 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 5988 T1121 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty1 forceNew: false
[junit4:junit4]   2> 5988 T1121 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty1/index/
[junit4:junit4]   2> 5988 T1121 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 5990 T1121 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty1/index forceNew: false
[junit4:junit4]   2> 6025 T1121 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050054tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@464d022d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 6025 T1121 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 6027 T1121 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 6028 T1121 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 6028 T1121 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 6029 T1121 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 6029 T1121 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 6029 T1121 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 6030 T1121 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 6030 T1121 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 6030 T1121 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 6037 T1121 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 6043 T1121 oass.SolrIndexSearcher.<init> Opening Searcher@12429c89 main
[junit4:junit4]   2> 6044 T1121 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty1/tlog
[junit4:junit4]   2> 6045 T1121 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 6045 T1121 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 6048 T1122 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@12429c89 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 6050 T1121 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 6050 T1121 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 6798 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 6799 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"2",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:36273_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:36273"}
[junit4:junit4]   2> 6799 T1101 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 2
[junit4:junit4]   2> 6800 T1101 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 6812 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6812 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 6812 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 7053 T1121 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 7053 T1121 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:36273 collection:collection1 shard:shard1
[junit4:junit4]   2> 7054 T1121 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 7073 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:delete cxid:0x48 zxid:0x5c txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders
[junit4:junit4]   2> 7074 T1121 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 7075 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:create cxid:0x49 zxid:0x5d txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 7084 T1121 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 7084 T1121 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 7084 T1121 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36273/collection1/
[junit4:junit4]   2> 7085 T1121 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 7085 T1121 oasc.SyncStrategy.syncToMe http://127.0.0.1:36273/collection1/ has no replicas
[junit4:junit4]   2> 7085 T1121 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36273/collection1/
[junit4:junit4]   2> 7085 T1121 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 7091 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:create cxid:0x53 zxid:0x61 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8319 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8363 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8363 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8363 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 8369 T1121 oasc.ZkController.register We are http://127.0.0.1:36273/collection1/ and leader is http://127.0.0.1:36273/collection1/
[junit4:junit4]   2> 8369 T1121 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:36273
[junit4:junit4]   2> 8369 T1121 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 8369 T1121 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 8370 T1121 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 8372 T1121 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 8373 T1078 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 8373 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 8374 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 8508 T1078 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 8511 T1078 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:16701
[junit4:junit4]   2> 8511 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 8512 T1078 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 8512 T1078 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005
[junit4:junit4]   2> 8512 T1078 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/solr.xml
[junit4:junit4]   2> 8512 T1078 oasc.CoreContainer.<init> New CoreContainer 1948911294
[junit4:junit4]   2> 8513 T1078 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/'
[junit4:junit4]   2> 8513 T1078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/'
[junit4:junit4]   2> 8547 T1078 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 8547 T1078 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 8547 T1078 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 8548 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 8548 T1078 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 8548 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 8549 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 8549 T1078 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 8549 T1078 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 8549 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 8565 T1078 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 8580 T1078 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:56885/solr
[junit4:junit4]   2> 8580 T1078 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 8581 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@73331176
[junit4:junit4]   2> 8582 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8582 T1133 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8583 T1133 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 8583 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:54776
[junit4:junit4]   2> 8584 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:54776
[junit4:junit4]   2> 8585 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20007 with negotiated timeout 20000 for client /140.211.11.196:54776
[junit4:junit4]   2> 8585 T1133 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20007, negotiated timeout = 20000
[junit4:junit4]   2> 8585 T1134 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73331176 name:ZooKeeperConnection Watcher:127.0.0.1:56885 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8585 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8587 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee20007
[junit4:junit4]   2> 8588 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:54776 which had sessionid 0x13d6f8dfee20007
[junit4:junit4]   2> 8588 T1134 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 8588 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee20007 closed
[junit4:junit4]   2> 8589 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 8596 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@550a98f4
[junit4:junit4]   2> 8597 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 8597 T1135 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 8598 T1135 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 8598 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:24068
[junit4:junit4]   2> 8598 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:24068
[junit4:junit4]   2> 8600 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20008 with negotiated timeout 20000 for client /140.211.11.196:24068
[junit4:junit4]   2> 8600 T1135 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20008, negotiated timeout = 20000
[junit4:junit4]   2> 8600 T1136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@550a98f4 name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 8600 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 8601 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20008 type:create cxid:0x1 zxid:0x6e txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8603 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20008 type:create cxid:0x2 zxid:0x6f txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 8612 T1078 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 9615 T1078 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:16701_
[junit4:junit4]   2> 9617 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20008 type:delete cxid:0xb zxid:0x70 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:16701_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:16701_
[junit4:junit4]   2> 9643 T1078 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:16701_
[junit4:junit4]   2> 9646 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9646 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 9647 T1136 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9647 T1107 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9647 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9649 T1100 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9650 T1120 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 9657 T1137 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/collection1
[junit4:junit4]   2> 9657 T1137 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 9658 T1137 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 9658 T1137 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 9660 T1137 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/collection1/'
[junit4:junit4]   2> 9661 T1137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/collection1/lib/README' to classloader
[junit4:junit4]   2> 9662 T1137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 9715 T1137 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 9783 T1137 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 9784 T1137 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 9791 T1137 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 9875 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 9876 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"2",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:36273_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:36273"}
[junit4:junit4]   2> 9934 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9934 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9934 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 9935 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 10448 T1137 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 10460 T1137 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 10464 T1137 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 10483 T1137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10488 T1137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 10492 T1137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10494 T1137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10494 T1137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10494 T1137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 10496 T1137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 10496 T1137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 10496 T1137 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10496 T1137 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1363376218005/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty2/
[junit4:junit4]   2> 10497 T1137 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11d13272
[junit4:junit4]   2> 10497 T1137 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 10498 T1137 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty2 forceNew: false
[junit4:junit4]   2> 10498 T1137 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty2/index/
[junit4:junit4]   2> 10498 T1137 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 10500 T1137 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty2/index forceNew: false
[junit4:junit4]   2> 10514 T1137 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050055tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@44ebdea4),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 10514 T1137 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 10517 T1137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 10517 T1137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 10518 T1137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 10518 T1137 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 10519 T1137 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 10519 T1137 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 10519 T1137 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 10520 T1137 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 10520 T1137 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 10526 T1137 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 10533 T1137 oass.SolrIndexSearcher.<init> Opening Searcher@1f42d526 main
[junit4:junit4]   2> 10533 T1137 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty2/tlog
[junit4:junit4]   2> 10534 T1137 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 10534 T1137 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 10538 T1138 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1f42d526 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 10539 T1137 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 10540 T1137 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 11469 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 11470 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:16701_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:16701"}
[junit4:junit4]   2> 11470 T1101 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 11470 T1101 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 11500 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11501 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11501 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11500 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 11542 T1137 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 11542 T1137 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:16701 collection:collection1 shard:shard2
[junit4:junit4]   2> 11543 T1137 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 11557 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20008 type:delete cxid:0x47 zxid:0x7f txntype:-1 reqpath:n/a Error Path:/solr/collections/collection1/leaders/shard2 Error:KeeperErrorCode = NoNode for /solr/collections/collection1/leaders/shard2
[junit4:junit4]   2> 11558 T1137 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 11559 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20008 type:create cxid:0x48 zxid:0x80 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 11561 T1137 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 11562 T1137 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 11562 T1137 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:16701/collection1/
[junit4:junit4]   2> 11562 T1137 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 11562 T1137 oasc.SyncStrategy.syncToMe http://127.0.0.1:16701/collection1/ has no replicas
[junit4:junit4]   2> 11563 T1137 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:16701/collection1/
[junit4:junit4]   2> 11563 T1137 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 11573 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20008 type:create cxid:0x51 zxid:0x83 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13006 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 13026 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13026 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13026 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13026 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 13055 T1137 oasc.ZkController.register We are http://127.0.0.1:16701/collection1/ and leader is http://127.0.0.1:16701/collection1/
[junit4:junit4]   2> 13055 T1137 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:16701
[junit4:junit4]   2> 13055 T1137 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 13055 T1137 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 13055 T1137 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 13057 T1137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 13059 T1078 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 13059 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 13060 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 13260 T1078 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 13264 T1078 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:58198
[junit4:junit4]   2> 13264 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 13265 T1078 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 13265 T1078 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696
[junit4:junit4]   2> 13266 T1078 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/solr.xml
[junit4:junit4]   2> 13266 T1078 oasc.CoreContainer.<init> New CoreContainer 682617955
[junit4:junit4]   2> 13267 T1078 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/'
[junit4:junit4]   2> 13267 T1078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/'
[junit4:junit4]   2> 13316 T1078 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 13316 T1078 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 13317 T1078 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 13317 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 13318 T1078 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 13318 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 13319 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 13319 T1078 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 13319 T1078 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 13320 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 13339 T1078 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 13362 T1078 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:56885/solr
[junit4:junit4]   2> 13363 T1078 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 13363 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3b0cb2cb
[junit4:junit4]   2> 13364 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13365 T1149 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13366 T1149 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 13366 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:58199
[junit4:junit4]   2> 13366 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:58199
[junit4:junit4]   2> 13368 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee20009 with negotiated timeout 20000 for client /140.211.11.196:58199
[junit4:junit4]   2> 13368 T1149 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee20009, negotiated timeout = 20000
[junit4:junit4]   2> 13368 T1150 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3b0cb2cb name:ZooKeeperConnection Watcher:127.0.0.1:56885 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13368 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13370 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee20009
[junit4:junit4]   2> 13371 T1150 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 13371 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:58199 which had sessionid 0x13d6f8dfee20009
[junit4:junit4]   2> 13371 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee20009 closed
[junit4:junit4]   2> 13372 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 13382 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@625480dc
[junit4:junit4]   2> 13383 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 13384 T1151 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 13384 T1151 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 13384 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:58200
[junit4:junit4]   2> 13385 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:58200
[junit4:junit4]   2> 13386 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee2000a with negotiated timeout 20000 for client /140.211.11.196:58200
[junit4:junit4]   2> 13386 T1151 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee2000a, negotiated timeout = 20000
[junit4:junit4]   2> 13386 T1152 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@625480dc name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 13386 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 13387 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee2000a type:create cxid:0x1 zxid:0x90 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13389 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee2000a type:create cxid:0x2 zxid:0x91 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 13398 T1078 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 14401 T1078 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58198_
[junit4:junit4]   2> 14402 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee2000a type:delete cxid:0xb zxid:0x92 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:58198_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:58198_
[junit4:junit4]   2> 14404 T1078 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58198_
[junit4:junit4]   2> 14407 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14407 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 14408 T1107 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14408 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14408 T1152 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14408 T1136 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14408 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14409 T1120 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14410 T1100 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 14422 T1153 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/collection1
[junit4:junit4]   2> 14423 T1153 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 14423 T1153 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 14423 T1153 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 14425 T1153 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/collection1/'
[junit4:junit4]   2> 14426 T1153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 14427 T1153 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/collection1/lib/README' to classloader
[junit4:junit4]   2> 14473 T1153 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 14529 T1153 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 14530 T1153 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 14533 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 14533 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"3",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:16701_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:16701"}
[junit4:junit4]   2> 14537 T1153 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 14546 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14546 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14546 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14546 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 14546 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 15152 T1153 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 15164 T1153 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 15169 T1153 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 15186 T1153 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15191 T1153 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 15195 T1153 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15197 T1153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15197 T1153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15198 T1153 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 15199 T1153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 15199 T1153 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 15199 T1153 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15200 T1153 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1363376222696/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3/
[junit4:junit4]   2> 15200 T1153 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11d13272
[junit4:junit4]   2> 15200 T1153 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 15201 T1153 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3 forceNew: false
[junit4:junit4]   2> 15201 T1153 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3/index/
[junit4:junit4]   2> 15201 T1153 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 15203 T1153 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3/index forceNew: false
[junit4:junit4]   2> 15208 T1153 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050056tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@64ae8431),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 15208 T1153 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 15211 T1153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 15211 T1153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 15212 T1153 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 15212 T1153 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 15213 T1153 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 15213 T1153 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 15213 T1153 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 15214 T1153 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 15214 T1153 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 15221 T1153 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 15228 T1153 oass.SolrIndexSearcher.<init> Opening Searcher@394db297 main
[junit4:junit4]   2> 15229 T1153 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3/tlog
[junit4:junit4]   2> 15230 T1153 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 15230 T1153 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 15233 T1154 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@394db297 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 15235 T1153 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 15235 T1153 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16054 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 16055 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:58198_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58198"}
[junit4:junit4]   2> 16055 T1101 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 16055 T1101 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 16068 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16068 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16068 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16068 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16068 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 16238 T1153 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 16238 T1153 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58198 collection:collection1 shard:shard1
[junit4:junit4]   2> 16241 T1153 oasc.ZkController.register We are http://127.0.0.1:58198/collection1/ and leader is http://127.0.0.1:36273/collection1/
[junit4:junit4]   2> 16242 T1153 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58198
[junit4:junit4]   2> 16242 T1153 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 16242 T1153 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C36 name=collection1 org.apache.solr.core.SolrCore@41c06ed8 url=http://127.0.0.1:58198/collection1 node=127.0.0.1:58198_ C36_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:58198_, base_url=http://127.0.0.1:58198}
[junit4:junit4]   2> 16249 T1155 C36 P58198 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 16249 T1153 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 16250 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 16250 T1155 C36 P58198 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 16250 T1155 C36 P58198 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 16251 T1078 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 16251 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 16252 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16252 T1155 C36 P58198 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 16458 T1078 oejs.Server.doStart jetty-8.1.8.v20121106
[junit4:junit4]   2> 16461 T1078 oejs.AbstractConnector.doStart Started SocketConnector@127.0.0.1:58204
[junit4:junit4]   2> 16462 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 16463 T1078 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 16463 T1078 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890
[junit4:junit4]   2> 16464 T1078 oasc.CoreContainer$Initializer.initialize looking for solr config file: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/solr.xml
[junit4:junit4]   2> 16464 T1078 oasc.CoreContainer.<init> New CoreContainer 1020187302
[junit4:junit4]   2> 16465 T1078 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/'
[junit4:junit4]   2> 16465 T1078 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/'
[junit4:junit4]   2> 16514 T1078 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 16514 T1078 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 16515 T1078 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 16515 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 16516 T1078 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 16516 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 16517 T1078 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 16517 T1078 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 16517 T1078 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 16518 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 16538 T1078 oasc.CoreContainer.load Registering Log Listener
[junit4:junit4]   2> 16556 T1078 oasc.CoreContainer.initZooKeeper Zookeeper client=127.0.0.1:56885/solr
[junit4:junit4]   2> 16556 T1078 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 16557 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885 sessionTimeout=60000 watcher=org.apache.solr.common.cloud.ConnectionManager@3747681
[junit4:junit4]   2> 16558 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16558 T1166 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16559 T1166 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 16559 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:58206
[junit4:junit4]   2> 16560 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:58206
[junit4:junit4]   2> 16561 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee2000b with negotiated timeout 20000 for client /140.211.11.196:58206
[junit4:junit4]   2> 16561 T1166 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee2000b, negotiated timeout = 20000
[junit4:junit4]   2> 16562 T1167 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3747681 name:ZooKeeperConnection Watcher:127.0.0.1:56885 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16562 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16564 T1083 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d6f8dfee2000b
[junit4:junit4]   2> 16573 T1167 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 16573 T1080 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:58206 which had sessionid 0x13d6f8dfee2000b
[junit4:junit4]   2> 16573 T1078 oaz.ZooKeeper.close Session: 0x13d6f8dfee2000b closed
[junit4:junit4]   2> 16574 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 16585 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885/solr sessionTimeout=30000 watcher=org.apache.solr.common.cloud.ConnectionManager@30bb8cc9
[junit4:junit4]   2> 16586 T1078 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 16587 T1168 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:56885. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 16587 T1168 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:56885, initiating session
[junit4:junit4]   2> 16588 T1080 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:58208
[junit4:junit4]   2> 16588 T1080 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:58208
[junit4:junit4]   2> 16590 T1082 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d6f8dfee2000c with negotiated timeout 20000 for client /140.211.11.196:58208
[junit4:junit4]   2> 16590 T1168 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:56885, sessionid = 0x13d6f8dfee2000c, negotiated timeout = 20000
[junit4:junit4]   2> 16590 T1169 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30bb8cc9 name:ZooKeeperConnection Watcher:127.0.0.1:56885/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 16590 T1078 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 16592 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee2000c type:create cxid:0x1 zxid:0xa3 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16593 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee2000c type:create cxid:0x2 zxid:0xa4 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 16602 T1078 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 17575 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 17576 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:58198_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58198"}
[junit4:junit4]   2> 17591 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17591 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17591 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17591 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17591 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17591 T1169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17605 T1078 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58204_
[junit4:junit4]   2> 17606 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee2000c type:delete cxid:0xd zxid:0xa9 txntype:-1 reqpath:n/a Error Path:/solr/live_nodes/127.0.0.1:58204_ Error:KeeperErrorCode = NoNode for /solr/live_nodes/127.0.0.1:58204_
[junit4:junit4]   2> 17607 T1078 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58204_
[junit4:junit4]   2> 17609 T1169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17610 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17610 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17610 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 17610 T1136 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17610 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17611 T1107 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17611 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 17612 T1120 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17612 T1152 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17612 T1100 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17613 T1169 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 17623 T1170 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/collection1
[junit4:junit4]   2> 17623 T1170 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 17624 T1170 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 17625 T1170 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 17626 T1170 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/collection1/'
[junit4:junit4]   2> 17627 T1170 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 17628 T1170 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/collection1/lib/README' to classloader
[junit4:junit4]   2> 17674 T1170 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 17729 T1170 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 17731 T1170 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 17737 T1170 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 18290 T1110 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=4&state=recovering&nodeName=127.0.0.1:58198_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2008
[junit4:junit4]   2> 18388 T1170 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 18402 T1170 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 18406 T1170 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 18424 T1170 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 18430 T1170 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 18435 T1170 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 18436 T1170 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 18437 T1170 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 18437 T1170 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 18438 T1170 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 18439 T1170 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 18439 T1170 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 18439 T1170 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1363376225890/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4/
[junit4:junit4]   2> 18439 T1170 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11d13272
[junit4:junit4]   2> 18440 T1170 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 18441 T1170 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4 forceNew: false
[junit4:junit4]   2> 18441 T1170 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4/index/
[junit4:junit4]   2> 18441 T1170 oasc.SolrCore.initIndex WARNING [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 18443 T1170 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4/index forceNew: false
[junit4:junit4]   2> 18449 T1170 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050057tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@6f43707d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 18449 T1170 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 18452 T1170 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 18452 T1170 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 18453 T1170 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 18453 T1170 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 18454 T1170 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 18454 T1170 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 18454 T1170 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 18455 T1170 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 18455 T1170 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 18463 T1170 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 18471 T1170 oass.SolrIndexSearcher.<init> Opening Searcher@5758f8e6 main
[junit4:junit4]   2> 18472 T1170 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4/tlog
[junit4:junit4]   2> 18473 T1170 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 18473 T1170 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 18477 T1171 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5758f8e6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 18479 T1170 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 18479 T1170 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19097 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 19098 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:58204_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58204"}
[junit4:junit4]   2> 19098 T1101 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
[junit4:junit4]   2> 19098 T1101 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 19111 T1169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19111 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19112 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19112 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19111 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19111 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 19481 T1170 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 19481 T1170 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58204 collection:collection1 shard:shard2
[junit4:junit4]   2> 19485 T1170 oasc.ZkController.register We are http://127.0.0.1:58204/collection1/ and leader is http://127.0.0.1:16701/collection1/
[junit4:junit4]   2> 19485 T1170 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58204
[junit4:junit4]   2> 19485 T1170 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 19485 T1170 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C37 name=collection1 org.apache.solr.core.SolrCore@26075b18 url=http://127.0.0.1:58204/collection1 node=127.0.0.1:58204_ C37_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, collection=collection1, node_name=127.0.0.1:58204_, base_url=http://127.0.0.1:58204}
[junit4:junit4]   2> 19486 T1172 C37 P58204 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 19486 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 19487 T1170 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 19487 T1172 C37 P58204 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 19487 T1172 C37 P58204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 19488 T1078 oass.SolrDispatchFilter.init user.dir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1
[junit4:junit4]   2> 19488 T1078 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 19489 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19499 T1172 C37 P58204 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19505 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 19506 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 19520 T1111 oasc.CoreContainer.create Creating SolrCore 'onenodecollectioncore' using instanceDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/onenodecollectioncore
[junit4:junit4]   2> 19520 T1111 oasc.ZkController.createCollectionZkNode Check for collection zkNode:onenodecollection
[junit4:junit4]   2> 19521 T1111 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:onenodecollection
[junit4:junit4]   2> 19521 T1111 oasc.ZkController.getConfName Looking for collection configName
[junit4:junit4]   2> 19523 T1111 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
[junit4:junit4]   2> 19523 T1111 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection
[junit4:junit4]   2> 19526 T1111 oasc.ZkController.readConfigName Load collection config from:/collections/onenodecollection
[junit4:junit4]   2> 19528 T1111 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/onenodecollectioncore/'
[junit4:junit4]   2> 19589 T1111 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_43
[junit4:junit4]   2> 19648 T1111 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 19650 T1111 oass.IndexSchema.readSchema Reading Solr Schema
[junit4:junit4]   2> 19656 T1111 oass.IndexSchema.readSchema Schema name=test
[junit4:junit4]   2> 20258 T1111 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 20270 T1111 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 20274 T1111 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2>  C36_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:58198_, base_url=http://127.0.0.1:58198}
[junit4:junit4]   2> 20292 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:36273/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 20297 T1111 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20302 T1155 C36 P58198 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 20304 T1111 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 20308 T1111 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 20310 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 20310 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 20311 T1111 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 20312 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 20313 T1111 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARNING Expected key, got STRING
[junit4:junit4]   2> 20313 T1111 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 20316 T1111 oasc.SolrCore.<init> [onenodecollectioncore] Opening new SolrCore at /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1363376213466/onenodecollectioncore/, dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection/
[junit4:junit4]   2> 20317 T1111 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@11d13272
[junit4:junit4]   2> 20317 T1111 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 20318 T1111 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection forceNew: false
[junit4:junit4]   2> 20318 T1111 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection/index/
[junit4:junit4]   2> 20319 T1111 oasc.SolrCore.initIndex WARNING [onenodecollectioncore] Solr index directory '/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 20320 T1155 C36 P58198 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58198 START replicas=[http://127.0.0.1:36273/collection1/] nUpdates=100
[junit4:junit4]   2> 20320 T1155 C36 P58198 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 20321 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 20321 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 20321 T1111 oasc.CachingDirectoryFactory.get return new directory for /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection/index forceNew: false
[junit4:junit4]   2> 20321 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 20322 T1155 C36 P58198 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 20322 T1155 C36 P58198 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:36273/collection1/. core=collection1
[junit4:junit4]   2> 20322 T1155 C36 P58198 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 20327 T1111 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050058tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@20a82c8d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20327 T1111 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> ASYNC  NEW_CORE C38 name=collection1 org.apache.solr.core.SolrCore@273b5b2a url=http://127.0.0.1:36273/collection1 node=127.0.0.1:36273_ C38_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:36273_, base_url=http://127.0.0.1:36273, leader=true}
[junit4:junit4]   2> 20328 T1112 C38 P36273 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 20330 T1111 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 20330 T1111 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 20331 T1111 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 20331 T1111 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 20332 T1111 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 20332 T1111 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 20333 T1111 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 20333 T1111 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 20334 T1111 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 20344 T1111 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 20357 T1111 oass.SolrIndexSearcher.<init> Opening Searcher@69a7f4b2 main
[junit4:junit4]   2> 20359 T1111 oasu.UpdateHandler.clearLog Clearing tlog files, tlogDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection/tlog
[junit4:junit4]   2> 20359 T1113 C38 P36273 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 20363 T1111 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 20364 T1111 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 20367 T1113 C38 P36273 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050054tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@464d022d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 20367 T1175 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@69a7f4b2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 20367 T1113 C38 P36273 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 20369 T1111 oasc.ZkController.publish publishing core=onenodecollectioncore state=down
[junit4:junit4]   2> 20373 T1113 C38 P36273 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050054tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@464d022d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050054tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@464d022d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20373 T1113 C38 P36273 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20374 T1113 C38 P36273 oass.SolrIndexSearcher.<init> Opening Searcher@4220f21b realtime
[junit4:junit4]   2> 20374 T1113 C38 P36273 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 20375 T1113 C38 P36273 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 36
[junit4:junit4]   2> 20376 T1155 C36 P58198 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20376 T1155 C36 P58198 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4]   2> 20402 T1155 C36 P58198 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 20404 T1114 C38 P36273 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 20404 T1114 C38 P36273 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4]   2> 20405 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 20406 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 20406 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 20409 T1114 C38 P36273 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=1
[junit4:junit4]   2> 20409 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 20410 T1155 C36 P58198 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty3/index.20130315183710032 forceNew: false
[junit4:junit4]   2> 20411 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@65fe1135 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fbdb173; maxCacheMB=0.4521484375 maxMergeSizeMB=0.5166015625)) fullCopy=false
[junit4:junit4]   2> 20424 T1114 C38 P36273 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=5
[junit4:junit4]   2> 20437 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 20446 T1155 C36 P58198 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 20446 T1155 C36 P58198 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 20450 T1155 C36 P58198 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050056tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@64ae8431),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050056tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@64ae8431),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 20450 T1155 C36 P58198 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 20451 T1155 C36 P58198 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 20451 T1155 C36 P58198 oass.SolrIndexSearcher.<init> Opening Searcher@305abda9 main
[junit4:junit4]   2> 20452 T1154 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@305abda9 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 20453 T1155 C36 P58198 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@65fe1135 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fbdb173; maxCacheMB=0.4521484375 maxMergeSizeMB=0.5166015625))
[junit4:junit4]   2> 20453 T1155 C36 P58198 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 20453 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 20453 T1155 C36 P58198 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 20453 T1155 C36 P58198 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 20455 T1155 C36 P58198 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 20618 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 20619 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:58204_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58204"}
[junit4:junit4]   2> 20623 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":null,
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":"none",
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"onenodecollectioncore",
[junit4:junit4]   2>  "collection":"onenodecollection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:36273_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:36273"}
[junit4:junit4]   2> 20623 T1101 oasc.Overseer$ClusterStateUpdater.createCollection Create collection onenodecollection with numShards 1
[junit4:junit4]   2> 20623 T1101 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 20635 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"4",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:58198_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58198"}
[junit4:junit4]   2> 20646 T1169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20646 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20647 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20646 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20646 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 20646 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 21373 T1111 oasc.CoreContainer.registerCore registering core: onenodecollectioncore
[junit4:junit4]   2> 21374 T1111 oasc.ZkController.register Register replica - core:onenodecollectioncore address:http://127.0.0.1:36273 collection:onenodecollection shard:shard1
[junit4:junit4]   2> 21375 T1111 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leader_elect/shard1/election
[junit4:junit4]   2> 21394 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:delete cxid:0xa8 zxid:0xc4 txntype:-1 reqpath:n/a Error Path:/solr/collections/onenodecollection/leaders Error:KeeperErrorCode = NoNode for /solr/collections/onenodecollection/leaders
[junit4:junit4]   2> 21395 T1111 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process.
[junit4:junit4]   2> 21396 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:create cxid:0xa9 zxid:0xc5 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 21398 T1111 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 21398 T1111 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 21399 T1111 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:36273/onenodecollectioncore/
[junit4:junit4]   2> 21399 T1111 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 21399 T1111 oasc.SyncStrategy.syncToMe http://127.0.0.1:36273/onenodecollectioncore/ has no replicas
[junit4:junit4]   2> 21400 T1111 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:36273/onenodecollectioncore/
[junit4:junit4]   2> 21400 T1111 oascc.SolrZkClient.makePath makePath: /collections/onenodecollection/leaders/shard1
[junit4:junit4]   2> 21405 T1083 oazs.PrepRequestProcessor.pRequest Got user-level KeeperException when processing sessionid:0x13d6f8dfee20006 type:create cxid:0xb3 zxid:0xc9 txntype:-1 reqpath:n/a Error Path:/solr/overseer Error:KeeperErrorCode = NodeExists for /solr/overseer
[junit4:junit4]   2> 21514 T1126 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {coreNodeName=5&state=recovering&nodeName=127.0.0.1:58204_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=2002
[junit4:junit4]   2> 22157 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 22209 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22209 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22209 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22209 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22209 T1169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22209 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 22223 T1111 oasc.ZkController.register We are http://127.0.0.1:36273/onenodecollectioncore/ and leader is http://127.0.0.1:36273/onenodecollectioncore/
[junit4:junit4]   2> 22223 T1111 oasc.ZkController.register No LogReplay needed for core=onenodecollectioncore baseURL=http://127.0.0.1:36273
[junit4:junit4]   2> 22223 T1111 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 22224 T1111 oasc.ZkController.publish publishing core=onenodecollectioncore state=active
[junit4:junit4]   2> 22226 T1111 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 22227 T1111 oass.SolrDispatchFilter.handleAdminRequest /admin/cores {numShards=1&roles=none&name=onenodecollectioncore&action=CREATE&collection=onenodecollection&dataDir=/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1363376209628/onenodecollection&wt=javabin&version=2} status=0 QTime=2708
[junit4:junit4]   2> 22227 T1078 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: onenodecollection failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 22229 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23231 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2>  C37_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=recovering, core=collection1, collection=collection1, node_name=127.0.0.1:58204_, base_url=http://127.0.0.1:58204}
[junit4:junit4]   2> 23515 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:16701/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 23515 T1172 C37 P58204 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58204 START replicas=[http://127.0.0.1:16701/collection1/] nUpdates=100
[junit4:junit4]   2> 23516 T1172 C37 P58204 oasu.PeerSync.sync WARNING no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 23516 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 23516 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 23517 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 23517 T1172 C37 P58204 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 23517 T1172 C37 P58204 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:16701/collection1/. core=collection1
[junit4:junit4]   2> 23518 T1172 C37 P58204 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C39 name=collection1 org.apache.solr.core.SolrCore@7b3aaba1 url=http://127.0.0.1:16701/collection1 node=127.0.0.1:16701_ C39_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:16701_, base_url=http://127.0.0.1:16701, leader=true}
[junit4:junit4]   2> 23526 T1127 C39 P16701 REQ /get {getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 23533 T1128 C39 P16701 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 23537 T1128 C39 P16701 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050055tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@44ebdea4),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 23538 T1128 C39 P16701 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 23545 T1128 C39 P16701 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050055tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@44ebdea4),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050055tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@44ebdea4),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23545 T1128 C39 P16701 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23546 T1128 C39 P16701 oass.SolrIndexSearcher.<init> Opening Searcher@1a73afed realtime
[junit4:junit4]   2> 23547 T1128 C39 P16701 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 23547 T1128 C39 P16701 /update {waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4]   2> 23548 T1172 C37 P58204 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23548 T1172 C37 P58204 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 23550 T1129 C39 P16701 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 23550 T1129 C39 P16701 REQ /replication {command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 23551 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex Master's generation: 2
[junit4:junit4]   2> 23551 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex Slave's generation: 1
[junit4:junit4]   2> 23551 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex Starting replication process
[junit4:junit4]   2> 23554 T1129 C39 P16701 REQ /replication {command=filelist&qt=/replication&wt=javabin&generation=2&version=2} status=0 QTime=0
[junit4:junit4]   2> 23555 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex Number of files in latest index in master: 1
[junit4:junit4]   2> 23556 T1172 C37 P58204 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1363376209627/jetty4/index.20130315183713177 forceNew: false
[junit4:junit4]   2> 23557 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex Starting download to MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@3ab6e92f lockFactory=org.apache.lucene.store.NativeFSLockFactory@54196399; maxCacheMB=0.4521484375 maxMergeSizeMB=0.5166015625)) fullCopy=false
[junit4:junit4]   2> 23559 T1129 C39 P16701 REQ /replication {file=segments_2&command=filecontent&checksum=true&wt=filestream&qt=/replication&generation=2} status=0 QTime=0
[junit4:junit4]   2> 23561 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex Total time taken for download : 0 secs
[junit4:junit4]   2> 23562 T1172 C37 P58204 oasu.DefaultSolrCoreState.newIndexWriter Creating new IndexWriter...
[junit4:junit4]   2> 23562 T1172 C37 P58204 oasu.DefaultSolrCoreState.newIndexWriter Waiting until IndexWriter is unused... core=collection1
[junit4:junit4]   2> 23565 T1172 C37 P58204 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050057tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@6f43707d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050057tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@6f43707d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> 23566 T1172 C37 P58204 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[segments_2]
[junit4:junit4]   2> 23566 T1172 C37 P58204 oasu.DefaultSolrCoreState.newIndexWriter New IndexWriter is ready to be used.
[junit4:junit4]   2> 23566 T1172 C37 P58204 oass.SolrIndexSearcher.<init> Opening Searcher@40819d8c main
[junit4:junit4]   2> 23567 T1171 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@40819d8c main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 23568 T1172 C37 P58204 oash.SnapPuller.fetchLatestIndex removing temporary index download directory files MockDirWrapper(NRTCachingDirectory(org.apache.lucene.store.RAMDirectory@3ab6e92f lockFactory=org.apache.lucene.store.NativeFSLockFactory@54196399; maxCacheMB=0.4521484375 maxMergeSizeMB=0.5166015625))
[junit4:junit4]   2> 23568 T1172 C37 P58204 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 23568 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 23568 T1172 C37 P58204 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 23569 T1172 C37 P58204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 23570 T1172 C37 P58204 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 23723 T1101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 23724 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"127.0.0.1:36273__onenodecollectioncore",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":"none",
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"onenodecollectioncore",
[junit4:junit4]   2>  "collection":"onenodecollection",
[junit4:junit4]   2>  "node_name":"127.0.0.1:36273_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:36273"}
[junit4:junit4]   2> 23728 T1101 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"5",
[junit4:junit4]   2>  "numShards":"2",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:58204_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58204"}
[junit4:junit4]   2> 23733 T1136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23733 T1152 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23733 T1120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23733 T1100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23733 T1107 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 23733 T1169 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 24235 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 24236 T1078 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: onenodecollection
[junit4:junit4]   2> 24236 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C40 name=onenodecollectioncore org.apache.solr.core.SolrCore@1f238d32 url=http://127.0.0.1:36273/onenodecollectioncore node=127.0.0.1:36273_ C40_STATE=coll:onenodecollection core:onenodecollectioncore props:{shard=shard1, roles=none, state=active, core=onenodecollectioncore, collection=onenodecollection, node_name=127.0.0.1:36273_, base_url=http://127.0.0.1:36273, leader=true}
[junit4:junit4]   2> 24267 T1115 C40 P36273 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050058tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@20a82c8d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24268 T1115 C40 P36273 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24271 T1115 C40 P36273 /update {wt=javabin&version=2} {add=[1 (1429603597822197760)]} 0 8
[junit4:junit4]   2> 24277 T1115 C40 P36273 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24334 T1115 C40 P36273 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050058tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@20a82c8d),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050058tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@20a82c8d),segFN=segments_2,generation=2,filenames=[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, segments_2, _0.fdx, _0.si, _0.prx, _0.fdt]
[junit4:junit4]   2> 24334 T1115 C40 P36273 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, segments_2, _0.fdx, _0.si, _0.prx, _0.fdt]
[junit4:junit4]   2> 24342 T1115 C40 P36273 oass.SolrIndexSearcher.<init> Opening Searcher@377d00d7 main
[junit4:junit4]   2> 24342 T1115 C40 P36273 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24343 T1175 oasc.SolrCore.registerSearcher [onenodecollectioncore] Registered new searcher Searcher@377d00d7 main{StandardDirectoryReader(segments_2:3 _0(4.3):C1)}
[junit4:junit4]   2> 24344 T1115 C40 P36273 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 67
[junit4:junit4]   2> 24347 T1115 C40 P36273 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4:junit4]   2> 24348 T1078 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 24366 T1115 C40 P36273 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4:junit4]   2> ASYNC  NEW_CORE C41 name=collection1 org.apache.solr.core.SolrCore@1cd6d763 url=http://127.0.0.1:14586/collection1 node=127.0.0.1:14586_ C41_STATE=coll:control_collection core:collection1 props:{shard=shard1, state=active, core=collection1, collection=control_collection, node_name=127.0.0.1:14586_, base_url=http://127.0.0.1:14586, leader=true}
[junit4:junit4]   2> 24389 T1090 C41 P14586 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits:num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050053tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bdf228e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> 24389 T1090 C41 P14586 oasc.SolrDeletionPolicy.updateCommits newest commit = 1[segments_1]
[junit4:junit4]   2> 24393 T1090 C41 P14586 /update {wt=javabin&version=2} {add=[1 (1429603597950124032)]} 0 9
[junit4:junit4]   2>  C36_STATE=coll:collection1 core:collection1 props:{shard=shard1, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:58198_, base_url=http://127.0.0.1:58198}
[junit4:junit4]   2> 24469 T1142 C36 P58198 /update {distrib.from=http://127.0.0.1:36273/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1429603598017232896)]} 0 5
[junit4:junit4]   2> 24470 T1116 C38 P36273 /update {distrib.from=http://127.0.0.1:16701/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1429603598017232896)]} 0 22
[junit4:junit4]   2> 24471 T1132 C39 P16701 /update {wt=javabin&version=2} {add=[1]} 0 66
[junit4:junit4]   2> 24473 T1090 C41 P14586 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24515 T1090 C41 P14586 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050053tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bdf228e),segFN=segments_1,generation=1,filenames=[segments_1]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050053tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bdf228e),segFN=segments_2,generation=2,filenames=[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, segments_2, _0.fdx, _0.si, _0.prx, _0.fdt]
[junit4:junit4]   2> 24515 T1090 C41 P14586 oasc.SolrDeletionPolicy.updateCommits newest commit = 2[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, segments_2, _0.fdx, _0.si, _0.prx, _0.fdt]
[junit4:junit4]   2> 24523 T1090 C41 P14586 oass.SolrIndexSearcher.<init> Opening Searcher@1672646a main
[junit4:junit4]   2> 24524 T1090 C41 P14586 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24524 T1104 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1672646a main{StandardDirectoryReader(segments_2:3 _0(4.3):C1)}
[junit4:junit4]   2> 24525 T1090 C41 P14586 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 52
[junit4:junit4]   2> 24527 T1110 C38 P36273 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24593 T1110 C38 P36273 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050054tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@464d022d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050054tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@464d022d),segFN=segments_3,generation=3,filenames=[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, _0.fdx, _0.si, _0.prx, _0.fdt, segments_3]
[junit4:junit4]   2> 24593 T1110 C38 P36273 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, _0.fdx, _0.si, _0.prx, _0.fdt, segments_3]
[junit4:junit4]   2> 24597 T1110 C38 P36273 oass.SolrIndexSearcher.<init> Opening Searcher@437af558 main
[junit4:junit4]   2> 24598 T1110 C38 P36273 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24598 T1122 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@437af558 main{StandardDirectoryReader(segments_3:3:nrt _0(4.3):C1)}
[junit4:junit4]   2> 24599 T1110 C38 P36273 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:58198/collection1/, StdNode: http://127.0.0.1:16701/collection1/, StdNode: http://127.0.0.1:58204/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 24601 T1126 C39 P16701 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24603 T1142 C36 P58198 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24606 T1126 C39 P16701 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050055tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@44ebdea4),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050055tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@44ebdea4),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 24606 T1126 C39 P16701 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 24607 T1126 C39 P16701 oass.SolrIndexSearcher.<init> Opening Searcher@7e35c4af main
[junit4:junit4]   2> 24607 T1126 C39 P16701 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24608 T1138 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7e35c4af main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24609 T1126 C39 P16701 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4]   2>  C37_STATE=coll:collection1 core:collection1 props:{shard=shard2, state=active, core=collection1, collection=collection1, node_name=127.0.0.1:58204_, base_url=http://127.0.0.1:58204}
[junit4:junit4]   2> 24613 T1159 C37 P58204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 24634 T1159 C37 P58204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050057tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@6f43707d),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050057tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@6f43707d),segFN=segments_3,generation=3,filenames=[segments_3]
[junit4:junit4]   2> 24634 T1159 C37 P58204 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[segments_3]
[junit4:junit4]   2> 24635 T1159 C37 P58204 oass.SolrIndexSearcher.<init> Opening Searcher@73258ce6 main
[junit4:junit4]   2> 24635 T1159 C37 P58204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24636 T1171 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@73258ce6 main{StandardDirectoryReader(segments_2:1:nrt)}
[junit4:junit4]   2> 24637 T1159 C37 P58204 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 24
[junit4:junit4]   2> 24659 T1142 C36 P58198 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits:num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050056tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@64ae8431),segFN=segments_2,generation=2,filenames=[segments_2]
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.NIOFSDirectory@/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J1/index5002050056tmp lockFactory=org.apache.lucene.store.NativeFSLockFactory@64ae8431),segFN=segments_3,generation=3,filenames=[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, _0.fdx, _0.si, _0.prx, _0.fdt, segments_3]
[junit4:junit4]   2> 24660 T1142 C36 P58198 oasc.SolrDeletionPolicy.updateCommits newest commit = 3[_0.tis, _0.nrm, _0.fnm, _0.tii, _0.frq, _0.fdx, _0.si, _0.prx, _0.fdt, segments_3]
[junit4:junit4]   2> 24665 T1142 C36 P58198 oass.SolrIndexSearcher.<init> Opening Searcher@4fb4cfab main
[junit4:junit4]   2> 24666 T1142 C36 P58198 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 24666 T1154 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4fb4cfab main{StandardDirectoryReader(segments_3:3:nrt _0(4.3):C1)}
[junit4:junit4]   2> 24667 T1142 C36 P58198 /update {waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 64
[junit4:junit4]   2> 24668 T1110 C38 P36273 /update {waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 141
[junit4:junit4]   2> 24668 T1078 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):230
[junit4:junit4]   2> 24669 T1078 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 24671 T1078 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 24674 T1090 C41 P14586 REQ /select {q=*:*&wt=javabin&version=2} hits=1 status=0 QTime=0
[junit4:junit4]   2> 24675 T1078 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:56885

[...truncated too long message...]

it4]   2> 121252 T1376 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty6/index.20130315154938479
[junit4:junit4]   2> 121252 T1376 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty6/index.20130315154938479
[junit4:junit4]   2> 121252 T1376 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2>
[junit4:junit4]   2> 121253 T1376 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 121253 T1376 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 121253 T1376 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 121253 T1376 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 122249 T1395 oasu.PeerSync.handleResponse WARNING PeerSync: core=collection1 url=http://127.0.0.1:56424/_sv/jv  couldn't connect to http://127.0.0.1:49094/_sv/jv/collection1/, counting as success
[junit4:junit4]   2> 122249 T1395 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:56424/_sv/jv DONE. sync succeeded
[junit4:junit4]   2> 122249 T1395 oasc.SyncStrategy.syncReplicas We have been closed, won't attempt to sync replicas back to leader
[junit4:junit4]   2> 122250 T1395 oasc.ShardLeaderElectionContext.rejoinLeaderElection Not rejoining election because CoreContainer is shutdown
[junit4:junit4]   2> 122250 T1395 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@6336f95a
[junit4:junit4]   2> 122251 T1395 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=4,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 122252 T1395 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 122252 T1395 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 122252 T1395 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2> 122253 T1395 C87 P56424 oasu.TransactionLog.forceClose SEVERE Error: Forcing close of tlog{file=./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7/tlog/tlog.0000000000000000003 refcount=1}
[junit4:junit4]   2> 122253 T1395 C87 P56424 oasu.TransactionLog.forceClose SEVERE Error: Forcing close of tlog{file=./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7/tlog/tlog.0000000000000000001 refcount=1}
[junit4:junit4]   2> 122256 T1395 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 122308 T1395 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7/index
[junit4:junit4]   2> 122308 T1395 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7/index
[junit4:junit4]   2> 122308 T1395 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7/index.20130315154942286
[junit4:junit4]   2> 122309 T1395 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7/index.20130315154942286
[junit4:junit4]   2> 122309 T1395 oasc.CachingDirectoryFactory.close Closing directory when closing factory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7
[junit4:junit4]   2> 122309 T1395 oasc.CachingDirectoryFactory.closeDirectory Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1363376949403/jetty7
[junit4:junit4]   2> 122309 T1395 oascc.ZkStateReader$3.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 122309 T1395 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 122310 T1395 oascc.ZkStateReader$2.process WARNING ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 122310 T1395 oasc.LeaderElector$1.process WARNING  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /overseer_elect/election
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
[junit4:junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
[junit4:junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
[junit4:junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
[junit4:junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
[junit4:junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
[junit4:junit4]   2>
[junit4:junit4]   2> 122310 T1395 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> NOTE: test params are: codec=Lucene41, sim=DefaultSimilarity, locale=it, timezone=America/St_Kitts
[junit4:junit4]   2> NOTE: FreeBSD 9.0-RELEASE amd64/Sun Microsystems Inc. 1.6.0_32 (64-bit)/cpus=16,threads=4,free=147357968,total=413532160
[junit4:junit4]   2> NOTE: All tests run in this JVM: [AlternateDirectoryTest, SynonymTokenizerTest, TestFastWriter, TestPhraseSuggestions, DocumentBuilderTest, SpatialFilterTest, TestQueryUtils, ResourceLoaderTest, FieldMutatingUpdateProcessorTest, SolrIndexSplitterTest, TestJoin, UpdateRequestProcessorFactoryTest, RAMDirectoryFactoryTest, RegexBoostProcessorTest, PreAnalyzedFieldTest, ResponseLogComponentTest, SystemInfoHandlerTest, SliceStateUpdateTest, TestGroupingSearch, TestCopyFieldCollectionResource, ZkCLITest, LoggingHandlerTest, SpellingQueryConverterTest, SliceStateTest, SuggesterFSTTest, QueryEqualityTest, SchemaVersionSpecificBehaviorTest, SOLR749Test, TestFuzzyAnalyzedSuggestions, TestPHPSerializedResponseWriter, DateFieldTest, TestFieldTypeCollectionResource, QueryElevationComponentTest, TestFaceting, CacheHeaderTest, UnloadDistributedZkTest, BasicFunctionalityTest, TestWriterPerf, CSVRequestHandlerTest, DirectSolrConnectionTest, SuggesterTest, TestRandomDVFaceting, DocValuesTest, TestLFUCache, ChaosMonkeyNothingIsSafeTest, TestReplicationHandler, SyncSliceTest]
[junit4:junit4] Completed on J0 in 122.70s, 1 test, 1 failure <<< FAILURES!

[...truncated 510 lines...]
BUILD FAILED
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:388: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:361: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/build.xml:39: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build.xml:183: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/common-build.xml:449: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:1213: The following error occurred while executing this line:
/usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/lucene/common-build.xml:877: There were test failures: 273 suites, 1147 tests, 1 error, 1 failure, 25 ignored (6 assumptions)

Total time: 110 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-NightlyTests-4.x - Build # 210 - Still Failing

Apache Jenkins Server-2
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-4.x/210/

1 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
No registered leader was found, collection:collection1 slice:shard1

Stack Trace:
org.apache.solr.common.SolrException: No registered leader was found, collection:collection1 slice:shard1
        at __randomizedtesting.SeedInfo.seed([D88F76030A38513A:5969F81B7D673106]:0)
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484)
        at org.apache.solr.cloud.BasicDistributedZk2Test.brindDownShardIndexSomeDocsAndRecover(BasicDistributedZk2Test.java:291)
        at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:116)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:806)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:679)




Build Log:
[...truncated 8791 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
[junit4:junit4]   2> 0 T1858 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /y/vv
[junit4:junit4]   2> 4 T1858 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./solrtest-BasicDistributedZk2Test-1363474885117
[junit4:junit4]   2> 5 T1858 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 5 T1859 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 7 T1859 oazs.ZooKeeperServer.setTickTime tickTime set to 1000
[junit4:junit4]   2> 7 T1859 oazs.ZooKeeperServer.setMinSessionTimeout minSessionTimeout set to -1
[junit4:junit4]   2> 7 T1859 oazs.ZooKeeperServer.setMaxSessionTimeout maxSessionTimeout set to -1
[junit4:junit4]   2> 8 T1859 oazs.NIOServerCnxnFactory.configure binding to port 0.0.0.0/0.0.0.0:0
[junit4:junit4]   2> 8 T1859 oazsp.FileTxnSnapLog.save Snapshotting: 0x0 to /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test/J0/./org.apache.solr.cloud.BasicDistributedZk2Test-1363474885117/zookeeper/server1/data/version-2/snapshot.0
[junit4:junit4]   2> 106 T1858 oasc.ZkTestServer.run start zk server on port:28415
[junit4:junit4]   2> 106 T1858 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:28415 sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@7d10228c
[junit4:junit4]   2> 107 T1858 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 107 T1864 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:28415. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 108 T1864 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:28415, initiating session
[junit4:junit4]   2> 108 T1860 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:28416
[junit4:junit4]   2> 108 T1860 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:28416
[junit4:junit4]   2> 109 T1862 oazsp.FileTxnLog.append Creating new log file: log.1
[junit4:junit4]   2> 114 T1862 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d756faa020000 with negotiated timeout 10000 for client /140.211.11.196:28416
[junit4:junit4]   2> 114 T1864 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:28415, sessionid = 0x13d756faa020000, negotiated timeout = 10000
[junit4:junit4]   2> 115 T1865 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7d10228c name:ZooKeeperConnection Watcher:127.0.0.1:28415 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 115 T1858 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 115 T1858 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 133 T1863 oazs.PrepRequestProcessor.pRequest2Txn Processed session termination for sessionid: 0x13d756faa020000
[junit4:junit4]   2> 134 T1865 oaz.ClientCnxn$EventThread.run EventThread shut down
[junit4:junit4]   2> 134 T1860 oazs.NIOServerCnxn.closeSock Closed socket connection for client /140.211.11.196:28416 which had sessionid 0x13d756faa020000
[junit4:junit4]   2> 134 T1858 oaz.ZooKeeper.close Session: 0x13d756faa020000 closed
[junit4:junit4]   2> 134 T1858 oaz.ZooKeeper.<init> Initiating client connection, connectString=127.0.0.1:28415/solr sessionTimeout=10000 watcher=org.apache.solr.common.cloud.ConnectionManager@699e4ea2
[junit4:junit4]   2> 135 T1858 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 136 T1866 oaz.ClientCnxn$SendThread.logStartConnect Opening socket connection to server localhost/127.0.0.1:28415. Will not attempt to authenticate using SASL (access denied (javax.security.auth.AuthPermission getLoginConfiguration))
[junit4:junit4]   2> 136 T1866 oaz.ClientCnxn$SendThread.primeConnection Socket connection established to localhost/127.0.0.1:28415, initiating session
[junit4:junit4]   2> 136 T1860 oazs.NIOServerCnxnFactory.run Accepted socket connection from /140.211.11.196:28417
[junit4:junit4]   2> 137 T1860 oazs.ZooKeeperServer.processConnectRequest Client attempting to establish new session at /140.211.11.196:28417
[junit4:junit4]   2> 151 T1862 oazs.ZooKeeperServer.finishSessionInit Established session 0x13d756faa020001 with negotiated timeout 10000 for client /140.211.11.196:28417
[junit4:junit4]   2> 151 T1866 oaz.ClientCnxn$SendThread.onConnected Session establishment complete on server localhost/127.0.0.1:28415, sessionid = 0x13d756faa020001, negotiated timeout = 10000
[junit4:junit4]   2> 151 T1867 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@699e4ea2 name:ZooKeeperConnection Watcher:127.0.0.1:28415/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 151 T1858 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 152 T1858 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 155 T1858 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 164 T1858 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 166 T1858 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 169 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 169 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 184 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 185 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 193 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 194 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 196 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 197 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 206 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 206 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 208 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 209 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 211 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 212 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 219 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 220 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 222 T1858 oasc.AbstractZkTestCase.putConfig put /usr/home/hudson/hudson-slave/workspace/Lucene-Solr-NightlyTests-4.x/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 223 T1858 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 225 T1