[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build # 6346 - Failure!

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build # 6346 - Failure!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6346/
Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:39944/sr/collection1lastClient and got 255 from http://127.0.0.1:50010/sr/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:39944/sr/collection1lastClient and got 255 from http://127.0.0.1:50010/sr/collection1
        at __randomizedtesting.SeedInfo.seed([C103F4B8ABD96C0D:40E57AA0DC860C31]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9660 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 664247 T2165 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /sr/
[junit4:junit4]   2> 664251 T2165 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372403597234
[junit4:junit4]   2> 664252 T2165 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 664252 T2166 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 664352 T2165 oasc.ZkTestServer.run start zk server on port:42941
[junit4:junit4]   2> 664354 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 664577 T2172 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4775b87d name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 664578 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 664578 T2165 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 664598 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 664600 T2174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15f44133 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 664601 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 664601 T2165 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 664606 T2165 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 664610 T2165 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 664614 T2165 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 664617 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 664618 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 664626 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 664626 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 664632 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 664632 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 664637 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 664638 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 664642 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 664642 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 664647 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 664648 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 664652 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 664653 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 664658 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 664658 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 664662 T2165 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 664663 T2165 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 664746 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 664748 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40343
[junit4:junit4]   2> 664749 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 664749 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 664749 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651
[junit4:junit4]   2> 664749 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/solr.xml
[junit4:junit4]   2> 664750 T2165 oasc.CoreContainer.<init> New CoreContainer 1935753536
[junit4:junit4]   2> 664750 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/'
[junit4:junit4]   2> 664750 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/'
[junit4:junit4]   2> 664818 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 664819 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 664819 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 664820 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 664821 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 664821 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 664822 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 664825 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 664826 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4]   2> 664826 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 664827 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 664828 T2185 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c8e6076 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 664829 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 664830 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 664834 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 664835 T2187 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@662e57d8 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 664835 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 664836 T2165 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 664839 T2165 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 664841 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 664843 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40343_sr
[junit4:junit4]   2> 664844 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40343_sr
[junit4:junit4]   2> 664847 T2165 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 664851 T2165 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 664854 T2165 oasc.Overseer.start Overseer (id=89941842148524035-127.0.0.1:40343_sr-n_0000000000) starting
[junit4:junit4]   2> 664860 T2165 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 664865 T2189 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 664865 T2165 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 664867 T2165 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 664869 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 664872 T2188 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 664874 T2190 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 664875 T2190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 664876 T2190 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 666375 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 666376 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:40343_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:40343/sr"}
[junit4:junit4]   2> 666376 T2188 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 666377 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 666381 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 666876 T2190 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 666877 T2190 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1
[junit4:junit4]   2> 666877 T2190 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 666878 T2190 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 666878 T2190 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 666880 T2190 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/'
[junit4:junit4]   2> 666881 T2190 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/lib/README' to classloader
[junit4:junit4]   2> 666881 T2190 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 666917 T2190 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 666958 T2190 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 666960 T2190 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 666964 T2190 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 667491 T2190 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 667496 T2190 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 667498 T2190 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 667534 T2190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 667537 T2190 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 667541 T2190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 667543 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 667544 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 667544 T2190 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 667546 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 667547 T2190 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 667547 T2190 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 667547 T2190 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372403597651/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/
[junit4:junit4]   2> 667548 T2190 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4]   2> 667549 T2190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data
[junit4:junit4]   2> 667550 T2190 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/index/
[junit4:junit4]   2> 667550 T2190 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 667551 T2190 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/control/data/index
[junit4:junit4]   2> 667555 T2190 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_1,generation=1}
[junit4:junit4]   2> 667556 T2190 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 667559 T2190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 667559 T2190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 667560 T2190 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 667560 T2190 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 667561 T2190 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 667561 T2190 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 667561 T2190 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 667562 T2190 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 667562 T2190 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 667566 T2190 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 667568 T2190 oass.SolrIndexSearcher.<init> Opening Searcher@7b2c6780 main
[junit4:junit4]   2> 667569 T2190 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 667569 T2190 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 667572 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b2c6780 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 667572 T2190 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 667573 T2190 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40343/sr collection:control_collection shard:shard1
[junit4:junit4]   2> 667573 T2190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 667579 T2190 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 667582 T2190 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 667582 T2190 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 667582 T2190 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40343/sr/collection1/
[junit4:junit4]   2> 667583 T2190 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 667583 T2190 oasc.SyncStrategy.syncToMe http://127.0.0.1:40343/sr/collection1/ has no replicas
[junit4:junit4]   2> 667583 T2190 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40343/sr/collection1/ shard1
[junit4:junit4]   2> 667584 T2190 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 667885 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 667893 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 667901 T2190 oasc.ZkController.register We are http://127.0.0.1:40343/sr/collection1/ and leader is http://127.0.0.1:40343/sr/collection1/
[junit4:junit4]   2> 667901 T2190 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40343/sr
[junit4:junit4]   2> 667901 T2190 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 667901 T2190 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 667902 T2190 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 667904 T2190 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 667915 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 667915 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 667916 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 667921 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 667926 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 667933 T2194 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b69efd2 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 667933 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 667934 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 667936 T2165 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 668041 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 668043 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39944
[junit4:junit4]   2> 668044 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 668045 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 668045 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918
[junit4:junit4]   2> 668046 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/solr.xml
[junit4:junit4]   2> 668046 T2165 oasc.CoreContainer.<init> New CoreContainer 528298614
[junit4:junit4]   2> 668046 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/'
[junit4:junit4]   2> 668047 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/'
[junit4:junit4]   2> 668091 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 668091 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 668092 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 668093 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 668093 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 668093 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 668094 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 668097 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 668097 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4]   2> 668098 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 668098 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 668099 T2205 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4a347ce4 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 668100 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 668101 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 668104 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 668105 T2207 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5dadfdec name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 668106 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 668108 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 669110 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39944_sr
[junit4:junit4]   2> 669112 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39944_sr
[junit4:junit4]   2> 669114 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 669115 T2207 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 669115 T2194 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 669119 T2187 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 669124 T2208 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 669124 T2208 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 669125 T2208 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 669397 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 669398 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:40343_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:40343/sr"}
[junit4:junit4]   2> 669401 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:39944_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4]   2> 669401 T2188 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 669401 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 669405 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 669405 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 669405 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 670126 T2208 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 670126 T2208 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1
[junit4:junit4]   2> 670126 T2208 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 670127 T2208 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 670127 T2208 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 670128 T2208 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/'
[junit4:junit4]   2> 670129 T2208 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/lib/README' to classloader
[junit4:junit4]   2> 670129 T2208 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 670163 T2208 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 670192 T2208 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 670193 T2208 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 670196 T2208 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 670530 T2208 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 670535 T2208 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 670537 T2208 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 670547 T2208 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 670550 T2208 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 670561 T2208 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 670562 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 670562 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 670563 T2208 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 670563 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 670564 T2208 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 670564 T2208 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 670564 T2208 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372403600918/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/
[junit4:junit4]   2> 670564 T2208 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4]   2> 670565 T2208 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1
[junit4:junit4]   2> 670565 T2208 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index/
[junit4:junit4]   2> 670565 T2208 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 670565 T2208 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index
[junit4:junit4]   2> 670567 T2208 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4]   2> 670567 T2208 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 670568 T2208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 670569 T2208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 670569 T2208 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 670570 T2208 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 670571 T2208 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 670571 T2208 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 670574 T2208 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 670576 T2208 oass.SolrIndexSearcher.<init> Opening Searcher@2bcf32cc main
[junit4:junit4]   2> 670577 T2208 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 670577 T2208 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 670580 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2bcf32cc main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 670581 T2208 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 670581 T2208 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39944/sr collection:collection1 shard:shard1
[junit4:junit4]   2> 670582 T2208 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 670587 T2208 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 670590 T2208 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 670590 T2208 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 670590 T2208 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 670590 T2208 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 670590 T2208 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/ has no replicas
[junit4:junit4]   2> 670590 T2208 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:39944/sr/collection1/ shard1
[junit4:junit4]   2> 670591 T2208 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 670910 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 670917 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 670917 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 670917 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 670946 T2208 oasc.ZkController.register We are http://127.0.0.1:39944/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 670946 T2208 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39944/sr
[junit4:junit4]   2> 670947 T2208 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 670947 T2208 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 670947 T2208 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 670949 T2208 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 670950 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 670950 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 670951 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 671016 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 671018 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53322
[junit4:junit4]   2> 671018 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 671019 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 671019 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936
[junit4:junit4]   2> 671019 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/solr.xml
[junit4:junit4]   2> 671019 T2165 oasc.CoreContainer.<init> New CoreContainer 461621668
[junit4:junit4]   2> 671020 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/'
[junit4:junit4]   2> 671020 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/'
[junit4:junit4]   2> 671066 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 671066 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 671066 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 671067 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 671067 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 671068 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 671068 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 671068 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 671069 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 671069 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 671072 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 671073 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4]   2> 671073 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 671074 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 671075 T2221 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5508c842 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 671076 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 671077 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 671082 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 671083 T2223 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7afd05c2 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 671083 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 671087 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 672091 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53322_sr
[junit4:junit4]   2> 672093 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53322_sr
[junit4:junit4]   2> 672097 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 672097 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 672098 T2194 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 672098 T2223 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 672098 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 672101 T2187 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 672101 T2207 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 672106 T2224 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 672106 T2224 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 672108 T2224 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 672424 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 672425 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:39944_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4]   2> 672430 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:53322_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4]   2> 672431 T2188 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 672431 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 672436 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 672436 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 672436 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 672436 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 673108 T2224 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 673109 T2224 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1
[junit4:junit4]   2> 673109 T2224 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 673110 T2224 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 673111 T2224 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 673113 T2224 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/'
[junit4:junit4]   2> 673114 T2224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/lib/README' to classloader
[junit4:junit4]   2> 673114 T2224 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 673154 T2224 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 673211 T2224 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 673213 T2224 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 673216 T2224 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 673597 T2224 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 673602 T2224 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 673604 T2224 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 673621 T2224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 673625 T2224 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 673629 T2224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 673631 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 673631 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 673632 T2224 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 673633 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 673634 T2224 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 673634 T2224 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 673634 T2224 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372403603936/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/
[junit4:junit4]   2> 673635 T2224 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4]   2> 673635 T2224 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2
[junit4:junit4]   2> 673636 T2224 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/index/
[junit4:junit4]   2> 673636 T2224 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 673636 T2224 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty2/index
[junit4:junit4]   2> 673639 T2224 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_1,generation=1}
[junit4:junit4]   2> 673639 T2224 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 673641 T2224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 673642 T2224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 673643 T2224 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 673643 T2224 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 673644 T2224 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 673644 T2224 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 673645 T2224 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 673646 T2224 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 673646 T2224 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 673650 T2224 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 673654 T2224 oass.SolrIndexSearcher.<init> Opening Searcher@46fc29ff main
[junit4:junit4]   2> 673655 T2224 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 673655 T2224 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 673662 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46fc29ff main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 673665 T2224 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 673665 T2224 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:53322/sr collection:collection1 shard:shard1
[junit4:junit4]   2> 673670 T2224 oasc.ZkController.register We are http://127.0.0.1:53322/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 673670 T2224 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:53322/sr
[junit4:junit4]   2> 673671 T2224 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 673671 T2224 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2456 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2456_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4]   2> 673672 T2226 C2456 P53322 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 673672 T2226 C2456 P53322 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 673673 T2224 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 673673 T2226 C2456 P53322 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 673673 T2226 C2456 P53322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 673675 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 673675 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 673675 T2226 C2456 P53322 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 673676 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 673687 T2203 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 673756 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 673758 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50010
[junit4:junit4]   2> 673758 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 673759 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 673759 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667
[junit4:junit4]   2> 673759 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/solr.xml
[junit4:junit4]   2> 673759 T2165 oasc.CoreContainer.<init> New CoreContainer 510334839
[junit4:junit4]   2> 673760 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/'
[junit4:junit4]   2> 673760 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/'
[junit4:junit4]   2> 673830 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 673831 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 673832 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 673833 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 673836 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 673836 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4]   2> 673836 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 673837 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 673839 T2238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6adb7cc3 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 673839 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 673841 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 673844 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 673846 T2240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@58130412 name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 673846 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 673849 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 673944 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 673946 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:53322_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4]   2> 673952 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 673952 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 673953 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 673952 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 673952 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 674688 T2203 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 674688 T2203 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:53322_sr&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 674852 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50010_sr
[junit4:junit4]   2> 674854 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50010_sr
[junit4:junit4]   2> 674857 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 674857 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 674857 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 674858 T2194 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 674858 T2240 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 674858 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 674859 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 674860 T2223 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 674861 T2207 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 674861 T2187 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 674867 T2241 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 674868 T2241 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 674869 T2241 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 675460 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 675462 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:50010_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4]   2> 675462 T2188 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 675462 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 675468 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 675469 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 675469 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 675468 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 675469 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 675870 T2241 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 675871 T2241 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1
[junit4:junit4]   2> 675871 T2241 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 675872 T2241 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 675872 T2241 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 675874 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/'
[junit4:junit4]   2> 675875 T2241 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/lib/README' to classloader
[junit4:junit4]   2> 675876 T2241 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 675904 T2241 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 675952 T2241 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 675953 T2241 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 675958 T2241 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 676370 T2241 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 676375 T2241 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 676378 T2241 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 676388 T2241 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 676391 T2241 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 676395 T2241 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 676396 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 676397 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 676397 T2241 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 676398 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 676398 T2241 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 676399 T2241 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 676399 T2241 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372403606667/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/
[junit4:junit4]   2> 676399 T2241 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4]   2> 676400 T2241 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3
[junit4:junit4]   2> 676400 T2241 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/index/
[junit4:junit4]   2> 676401 T2241 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 676401 T2241 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty3/index
[junit4:junit4]   2> 676404 T2241 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 676404 T2241 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 676407 T2241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 676407 T2241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 676408 T2241 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 676409 T2241 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 676410 T2241 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 676411 T2241 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 676411 T2241 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 676412 T2241 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 676413 T2241 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 676418 T2241 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 676421 T2241 oass.SolrIndexSearcher.<init> Opening Searcher@4379d712 main
[junit4:junit4]   2> 676421 T2241 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 676422 T2241 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 676425 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4379d712 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 676426 T2241 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 676426 T2241 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50010/sr collection:collection1 shard:shard1
[junit4:junit4]   2> 676430 T2241 oasc.ZkController.register We are http://127.0.0.1:50010/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 676430 T2241 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50010/sr
[junit4:junit4]   2> 676430 T2241 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 676430 T2241 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2457 name=collection1 org.apache.solr.core.SolrCore@256efb38 url=http://127.0.0.1:50010/sr/collection1 node=127.0.0.1:50010_sr C2457_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4]   2> 676431 T2243 C2457 P50010 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 676431 T2241 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 676431 T2243 C2457 P50010 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 676432 T2243 C2457 P50010 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 676432 T2243 C2457 P50010 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 676433 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 676434 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 676433 T2243 C2457 P50010 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 676434 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 676439 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 676528 T2165 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 676530 T2165 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:39475
[junit4:junit4]   2> 676531 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 676531 T2165 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 676531 T2165 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420
[junit4:junit4]   2> 676532 T2165 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/solr.xml
[junit4:junit4]   2> 676532 T2165 oasc.CoreContainer.<init> New CoreContainer 876879168
[junit4:junit4]   2> 676532 T2165 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/'
[junit4:junit4]   2> 676532 T2165 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/'
[junit4:junit4]   2> 676592 T2165 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 676592 T2165 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 676592 T2165 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 676593 T2165 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 676594 T2165 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 676594 T2165 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 676594 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 676598 T2165 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 676599 T2165 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:42941/solr
[junit4:junit4]   2> 676599 T2165 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 676600 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 676601 T2255 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@781a5d46 name:ZooKeeperConnection Watcher:127.0.0.1:42941 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 676602 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 676603 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 676606 T2165 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 676607 T2257 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6e78a7d name:ZooKeeperConnection Watcher:127.0.0.1:42941/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 676607 T2165 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 676610 T2165 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C2458 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2458_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4]   2> 676690 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39944/sr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 676690 T2226 C2458 P53322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 676690 T2226 C2458 P53322 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 676691 T2226 C2458 P53322 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 676691 T2226 C2458 P53322 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39944/sr/collection1/. core=collection1
[junit4:junit4]   2> 676691 T2226 C2458 P53322 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C2459 name=collection1 org.apache.solr.core.SolrCore@35d6ca2b url=http://127.0.0.1:39944/sr/collection1 node=127.0.0.1:39944_sr C2459_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4]   2> 676697 T2199 C2459 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 676700 T2201 C2459 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 676702 T2201 C2459 P39944 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4]   2> 676703 T2201 C2459 P39944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 676703 T2201 C2459 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 676703 T2201 C2459 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@6aa257e5 realtime
[junit4:junit4]   2> 676703 T2201 C2459 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 676704 T2201 C2459 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 676705 T2226 C2458 P53322 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 676705 T2226 C2458 P53322 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 676706 T2203 C2459 P39944 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 676706 T2203 C2459 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 676707 T2226 C2458 P53322 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 676707 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 676707 T2226 C2458 P53322 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 676707 T2226 C2458 P53322 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 676708 T2226 C2458 P53322 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 676976 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 676977 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:50010_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4]   2> 676982 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:53322_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4]   2> 676990 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 676990 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 676990 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 676990 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 676990 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 676990 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 677439 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 677440 T2202 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:50010_sr&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 677613 T2165 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:39475_sr
[junit4:junit4]   2> 677615 T2165 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:39475_sr
[junit4:junit4]   2> 677618 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 677619 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 677619 T2240 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 677618 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 677618 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 677619 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 677619 T2194 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 677621 T2223 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 677621 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 677622 T2187 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 677623 T2257 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 677623 T2207 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 677630 T2259 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 677630 T2259 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 677633 T2259 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 678500 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 678501 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:39475_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4]   2> 678501 T2188 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 678502 T2188 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 678508 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 678508 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 678508 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 678508 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 678508 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 678508 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 678634 T2259 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 678634 T2259 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1
[junit4:junit4]   2> 678634 T2259 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 678635 T2259 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 678636 T2259 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 678638 T2259 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/'
[junit4:junit4]   2> 678639 T2259 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/lib/README' to classloader
[junit4:junit4]   2> 678639 T2259 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 678682 T2259 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 678711 T2259 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 678713 T2259 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 678716 T2259 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 679098 T2259 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 679103 T2259 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 679105 T2259 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 679123 T2259 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 679128 T2259 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 679133 T2259 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 679135 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 679135 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 679135 T2259 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 679137 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 679138 T2259 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 679138 T2259 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 679138 T2259 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372403609420/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/
[junit4:junit4]   2> 679139 T2259 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@3b71c166
[junit4:junit4]   2> 679140 T2259 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4
[junit4:junit4]   2> 679140 T2259 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/index/
[junit4:junit4]   2> 679141 T2259 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 679141 T2259 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty4/index
[junit4:junit4]   2> 679143 T2259 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_1,generation=1}
[junit4:junit4]   2> 679144 T2259 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 679146 T2259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 679147 T2259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 679148 T2259 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 679149 T2259 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 679150 T2259 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 679150 T2259 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 679150 T2259 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 679151 T2259 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 679152 T2259 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 679156 T2259 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 679160 T2259 oass.SolrIndexSearcher.<init> Opening Searcher@10aee96d main
[junit4:junit4]   2> 679160 T2259 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 679160 T2259 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 679164 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10aee96d main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 679165 T2259 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 679165 T2259 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:39475/sr collection:collection1 shard:shard1
[junit4:junit4]   2> 679169 T2259 oasc.ZkController.register We are http://127.0.0.1:39475/sr/collection1/ and leader is http://127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 679169 T2259 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:39475/sr
[junit4:junit4]   2> 679169 T2259 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 679169 T2259 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C2460 name=collection1 org.apache.solr.core.SolrCore@7056e8b4 url=http://127.0.0.1:39475/sr/collection1 node=127.0.0.1:39475_sr C2460_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4]   2> 679170 T2261 C2460 P39475 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 679171 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 679171 T2259 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 679171 T2261 C2460 P39475 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 679172 T2261 C2460 P39475 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 679173 T2165 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 679173 T2261 C2460 P39475 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 679173 T2165 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 679174 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 679181 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 679182 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 679183 T2165 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4]   2> 679183 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4]   2> 679184 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C2461 name=collection1 org.apache.solr.core.SolrCore@256efb38 url=http://127.0.0.1:50010/sr/collection1 node=127.0.0.1:50010_sr C2461_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4]   2> 679441 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39944/sr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 679441 T2243 C2461 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 679442 T2243 C2461 P50010 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 679443 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 679444 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 679444 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 679444 T2243 C2461 P50010 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 679444 T2243 C2461 P50010 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39944/sr/collection1/. core=collection1
[junit4:junit4]   2> 679445 T2243 C2461 P50010 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C2462 name=collection1 org.apache.solr.core.SolrCore@35d6ca2b url=http://127.0.0.1:39944/sr/collection1 node=127.0.0.1:39944_sr C2462_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4]   2> 679445 T2203 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 679451 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 679452 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 679452 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 679452 T2201 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4]   2> 679454 T2243 C2461 P50010 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 679454 T2243 C2461 P50010 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 679456 T2203 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 679457 T2243 C2461 P50010 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 679457 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 679457 T2243 C2461 P50010 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 679458 T2243 C2461 P50010 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 679459 T2243 C2461 P50010 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 680014 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 680015 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:39475_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4]   2> 680019 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:50010_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4]   2> 680022 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 680022 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 680022 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 680022 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 680022 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 680022 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 680182 T2202 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 680183 T2202 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:39475_sr&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 680186 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 681189 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2>  C2460_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4]   2> 682184 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:39944/sr/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 682185 T2261 C2460 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 682185 T2261 C2460 P39475 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 682186 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 682186 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 682187 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 682187 T2261 C2460 P39475 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 682187 T2261 C2460 P39475 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:39944/sr/collection1/. core=collection1
[junit4:junit4]   2> 682188 T2261 C2460 P39475 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 682188 T2203 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 682191 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 682195 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 682195 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 682196 T2201 C2462 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 682196 T2201 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4]   2> 682197 T2261 C2460 P39475 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 682197 T2261 C2460 P39475 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 682200 T2202 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 682201 T2261 C2460 P39475 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 682201 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 682201 T2261 C2460 P39475 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 682201 T2261 C2460 P39475 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 682203 T2261 C2460 P39475 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 683032 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 683034 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:39475_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4]   2> 683041 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 683041 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 683041 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 683041 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 683041 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 683041 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 683194 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 683196 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C2463 name=collection1 org.apache.solr.core.SolrCore@547e62a7 url=http://127.0.0.1:40343/sr/collection1 node=127.0.0.1:40343_sr C2463_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40343_sr, base_url=http://127.0.0.1:40343/sr, leader=true}
[junit4:junit4]   2> 683206 T2180 C2463 P40343 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 683209 T2180 C2463 P40343 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_1,generation=1}
[junit4:junit4]   2> 683210 T2180 C2463 P40343 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 683211 T2180 C2463 P40343 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 683212 T2180 C2463 P40343 oass.SolrIndexSearcher.<init> Opening Searcher@6cdd19d7 main
[junit4:junit4]   2> 683212 T2180 C2463 P40343 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 683213 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6cdd19d7 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 683214 T2180 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4]   2>  C2461_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4]   2> 683227 T2236 C2461 P50010 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39944/sr/collection1/, StdNode: http://127.0.0.1:53322/sr/collection1/, StdNode: http://127.0.0.1:50010/sr/collection1/, StdNode: http://127.0.0.1:39475/sr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 683232 T2203 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 683233 T2203 C2462 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 683234 T2203 C2462 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@7654ebb1 main
[junit4:junit4]   2> 683234 T2203 C2462 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 683235 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7654ebb1 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 683237 T2234 C2461 P50010 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 683237 T2203 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 683240 T2234 C2461 P50010 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 683241 T2234 C2461 P50010 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 683242 T2234 C2461 P50010 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> ASYNC  NEW_CORE C2464 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2464_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4]   2> 683242 T2217 C2464 P53322 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 683243 T2234 C2461 P50010 oass.SolrIndexSearcher.<init> Opening Searcher@1b3ecbfb main
[junit4:junit4]   2> 683244 T2234 C2461 P50010 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2>  C2460_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4]   2> 683246 T2250 C2460 P39475 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 683246 T2217 C2464 P53322 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_1,generation=1}
[junit4:junit4]   2> 683246 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1b3ecbfb main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 683247 T2217 C2464 P53322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 683248 T2234 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4]   2> 683248 T2217 C2464 P53322 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 683248 T2250 C2460 P39475 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_1,generation=1}
[junit4:junit4]   2> 683250 T2217 C2464 P53322 oass.SolrIndexSearcher.<init> Opening Searcher@6dcdfe8 main
[junit4:junit4]   2> 683251 T2250 C2460 P39475 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 683251 T2217 C2464 P53322 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 683252 T2250 C2460 P39475 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 683252 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6dcdfe8 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 683253 T2250 C2460 P39475 oass.SolrIndexSearcher.<init> Opening Searcher@5f488d27 main
[junit4:junit4]   2> 683253 T2217 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 11
[junit4:junit4]   2> 683253 T2250 C2460 P39475 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 683255 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5f488d27 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 683255 T2250 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 10
[junit4:junit4]   2> 683256 T2236 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4]   2> 683258 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 683261 T2201 C2462 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4]   2> 683264 T2218 C2464 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 683267 T2235 C2461 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 683269 T2251 C2460 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4]   2> 685275 T2183 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439069496415354880)} 0 2
[junit4:junit4]   2> 685285 T2250 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=FROMLEADER&_version_=-1439069496422694912&update.from=http://127.0.0.1:39944/sr/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 1
[junit4:junit4]   2> 685286 T2217 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=FROMLEADER&_version_=-1439069496422694912&update.from=http://127.0.0.1:39944/sr/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 2
[junit4:junit4]   2> 685286 T2233 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=FROMLEADER&_version_=-1439069496422694912&update.from=http://127.0.0.1:39944/sr/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 2
[junit4:junit4]   2> 685287 T2203 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439069496422694912)} 0 7
[junit4:junit4]   2> 685288 T2236 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 11
[junit4:junit4]   2> 685294 T2181 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[0 (1439069496434229248)]} 0 3
[junit4:junit4]   2> 685305 T2216 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 2
[junit4:junit4]   2> 685305 T2234 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 2
[junit4:junit4]   2> 685305 T2251 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 2
[junit4:junit4]   2> 685307 T2199 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:53322/sr/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439069496442617856)]} 0 8
[junit4:junit4]   2> 685308 T2219 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[0]} 0 12
[junit4:junit4]   2> 685312 T2182 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[1 (1439069496455200768)]} 0 1
[junit4:junit4]   2> 685323 T2233 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 2
[junit4:junit4]   2> 685323 T2252 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 2
[junit4:junit4]   2> 685323 T2217 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 2
[junit4:junit4]   2> 685324 T2202 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:50010/sr/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1439069496462540800)]} 0 7
[junit4:junit4]   2> 685325 T2235 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[1]} 0 10
[junit4:junit4]   2> 685329 T2180 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[2 (1439069496473026560)]} 0 1
[junit4:junit4]   2> 685335 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:53322/sr/collection1/]
[junit4:junit4]   2> 685336 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:53322/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:true
[junit4:junit4]   2> 685336 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50010/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4]   2> 685337 T2201 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39475/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4]   2> 685342 T2236 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2 (1439069496482463744)]} 0 1
[junit4:junit4]   2> 685342 T2253 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2 (1439069496482463744)]} 0 1
[junit4:junit4]   2> 685344 T2201 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:53322/sr/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2 (1439069496482463744)]} 0 9
[junit4:junit4]   2> 685345 T2218 C2464 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[2]} 0 13
[junit4:junit4]   2> 685349 T2183 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[3 (1439069496493998080)]} 0 2
[junit4:junit4]   2> 685354 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:53322/sr/collection1/]
[junit4:junit4]   2> 685355 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:53322/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:true
[junit4:junit4]   2> 685355 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50010/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4]   2> 685356 T2203 C2462 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39475/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4]   2> 685361 T2233 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[3 (1439069496502386688)]} 0 2
[junit4:junit4]   2> 685361 T2250 C2460 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[3 (1439069496502386688)]} 0 2
[junit4:junit4]   2> 685362 T2203 C2462 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:50010/sr/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[3 (1439069496502386688)]} 0 8
[junit4:junit4]   2> 685363 T2234 C2461 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&test.distrib.skip.servers=http://127.0.0.1:50010/sr/collection1/&wt=javabin&version=2} {add=[3]} 0 11
[junit4:junit4]   2> 685365 T2181 C2463 P40343 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685369 T2181 C2463 P40343 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@599cdbaf lockFactory=org.apache.lucene.store.NativeFSLockFactory@475f075f),segFN=segments_2,generation=2}
[junit4:junit4]   2> 685370 T2181 C2463 P40343 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 685372 T2181 C2463 P40343 oass.SolrIndexSearcher.<init> Opening Searcher@6e7780f0 main
[junit4:junit4]   2> 685373 T2181 C2463 P40343 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685374 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6e7780f0 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685374 T2181 C2463 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 685377 T2199 C2462 P39944 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39944/sr/collection1/, StdNode: http://127.0.0.1:53322/sr/collection1/, StdNode: http://127.0.0.1:50010/sr/collection1/, StdNode: http://127.0.0.1:39475/sr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 685379 T2202 C2462 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685380 T2251 C2460 P39475 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685379 T2235 C2461 P50010 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685379 T2216 C2464 P53322 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685385 T2251 C2460 P39475 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@71b09f32 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6c5224d1),segFN=segments_2,generation=2}
[junit4:junit4]   2> ASYNC  NEW_CORE C2465 name=collection1 org.apache.solr.core.SolrCore@7056e8b4 url=http://127.0.0.1:39475/sr/collection1 node=127.0.0.1:39475_sr C2465_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39475_sr, base_url=http://127.0.0.1:39475/sr}
[junit4:junit4]   2> 685405 T2251 C2465 P39475 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> ASYNC  NEW_CORE C2466 name=collection1 org.apache.solr.core.SolrCore@256efb38 url=http://127.0.0.1:50010/sr/collection1 node=127.0.0.1:50010_sr C2466_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:50010_sr, base_url=http://127.0.0.1:50010/sr}
[junit4:junit4]   2> 685406 T2235 C2466 P50010 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2d73c5d3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7c39811e),segFN=segments_2,generation=2}
[junit4:junit4]   2> ASYNC  NEW_CORE C2467 name=collection1 org.apache.solr.core.SolrCore@35d6ca2b url=http://127.0.0.1:39944/sr/collection1 node=127.0.0.1:39944_sr C2467_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4]   2> 685406 T2202 C2467 P39944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_2,generation=2}
[junit4:junit4]   2> 685407 T2251 C2465 P39475 oass.SolrIndexSearcher.<init> Opening Searcher@29b5bc49 main
[junit4:junit4]   2> 685407 T2202 C2467 P39944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 685407 T2235 C2466 P50010 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> ASYNC  NEW_CORE C2468 name=collection1 org.apache.solr.core.SolrCore@5d541d9e url=http://127.0.0.1:53322/sr/collection1 node=127.0.0.1:53322_sr C2468_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:53322_sr, base_url=http://127.0.0.1:53322/sr}
[junit4:junit4]   2> 685406 T2216 C2468 P53322 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_2,generation=2}
[junit4:junit4]   2> 685408 T2216 C2468 P53322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 685408 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@29b5bc49 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685407 T2251 C2465 P39475 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685409 T2251 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4]   2> 685408 T2235 C2466 P50010 oass.SolrIndexSearcher.<init> Opening Searcher@3fb589b1 main
[junit4:junit4]   2> 685408 T2202 C2467 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@34bc6922 main
[junit4:junit4]   2> 685411 T2235 C2466 P50010 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685410 T2216 C2468 P53322 oass.SolrIndexSearcher.<init> Opening Searcher@67edd42c main
[junit4:junit4]   2> 685411 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3fb589b1 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685411 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685412 T2235 C2466 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4]   2> 685411 T2216 C2468 P53322 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685411 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34bc6922 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685412 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@67edd42c main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 685413 T2202 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4]   2> 685413 T2216 C2468 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4]   2> 685413 T2199 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 36
[junit4:junit4]   2> 685414 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4]   2> 685414 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 685415 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 685418 T2201 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 685420 T2219 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4]   2> 685422 T2236 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 685425 T2252 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 685425 T2165 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 685431 T2233 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 685431 T2233 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 685437 T2203 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 685438 T2203 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 685438 T2203 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39944/sr START replicas=[http://127.0.0.1:53322/sr/collection1/, http://127.0.0.1:50010/sr/collection1/, http://127.0.0.1:39475/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 685440 T2217 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685440 T2250 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685440 T2234 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685441 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr  Received 3 versions from 127.0.0.1:53322/sr/collection1/
[junit4:junit4]   2> 685442 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr  Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496462540800
[junit4:junit4]   2> 685442 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr  Received 5 versions from 127.0.0.1:39475/sr/collection1/
[junit4:junit4]   2> 685443 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr  Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4]   2> 685443 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr  Received 5 versions from 127.0.0.1:50010/sr/collection1/
[junit4:junit4]   2> 685443 T2203 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39944/sr  Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4]   2> 685444 T2203 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39944/sr DONE. sync succeeded
[junit4:junit4]   2> 685444 T2203 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 685445 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: try and ask http://127.0.0.1:53322/sr/collection1/ to sync
[junit4:junit4]   2> 685445 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: try and ask http://127.0.0.1:50010/sr/collection1/ to sync
[junit4:junit4]   2> 685446 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/: try and ask http://127.0.0.1:39475/sr/collection1/ to sync
[junit4:junit4]   2> 685447 T2218 C2468 P53322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 685447 T2235 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 685448 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr START replicas=[http://127.0.0.1:39944/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 685449 T2199 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685449 T2202 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685450 T2218 C2468 P53322 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr  Received 5 versions from 127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 685450 T2201 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685450 T2235 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr  Received 5 versions from 127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 685451 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr  Received 5 versions from 127.0.0.1:39944/sr/collection1/
[junit4:junit4]   2> 685452 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr  Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4]   2> 685451 T2218 C2468 P53322 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Requesting updates from 127.0.0.1:39944/sr/collection1/n=2 versions=[1439069496502386688, 1439069496482463744]
[junit4:junit4]   2> 685452 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr DONE. sync succeeded
[junit4:junit4]   2> 685453 T2253 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:39944/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=5
[junit4:junit4]   2> 685451 T2235 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr  Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496482463744
[junit4:junit4]   2> 685454 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/:  sync completed with http://127.0.0.1:39475/sr/collection1/
[junit4:junit4]   2> 685454 T2235 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr DONE. sync succeeded
[junit4:junit4]   2> 685455 T2199 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={distrib=false&getUpdates=1439069496502386688,1439069496482463744&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 685455 T2235 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:39944/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4]   2> 685456 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/:  sync completed with http://127.0.0.1:50010/sr/collection1/
[junit4:junit4]   2> 685459 T2218 C2468 P53322 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439069496482463744), 3 (1439069496502386688)]} 0 3
[junit4:junit4]   2> 685459 T2218 C2468 P53322 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr DONE. sync succeeded
[junit4:junit4]   2> 685460 T2218 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:39944/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=14
[junit4:junit4]   2> 685461 T2203 oasc.SyncStrategy.syncToMe http://127.0.0.1:39944/sr/collection1/:  sync completed with http://127.0.0.1:53322/sr/collection1/
[junit4:junit4]   2> 685461 T2203 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=24
[junit4:junit4]   2> 685462 T2233 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=31
[junit4:junit4]   2> 685463 T2165 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 685464 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 685465 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 685466 T2165 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C2469 name=collection1 org.apache.solr.core.SolrCore@547e62a7 url=http://127.0.0.1:40343/sr/collection1 node=127.0.0.1:40343_sr C2469_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40343_sr, base_url=http://127.0.0.1:40343/sr, leader=true}
[junit4:junit4]   2> 685469 T2182 C2469 P40343 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685470 T2182 C2469 P40343 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 685470 T2182 C2469 P40343 oass.SolrIndexSearcher.<init> Opening Searcher@1dfbe8df main
[junit4:junit4]   2> 685471 T2182 C2469 P40343 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685472 T2191 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1dfbe8df main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685472 T2182 C2469 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 685475 T2252 C2465 P39475 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:39944/sr/collection1/, StdNode: http://127.0.0.1:53322/sr/collection1/, StdNode: http://127.0.0.1:50010/sr/collection1/, StdNode: http://127.0.0.1:39475/sr/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 685479 T2236 C2466 P50010 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685479 T2250 C2465 P39475 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685480 T2236 C2466 P50010 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 685479 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685479 T2216 C2468 P53322 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 685482 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 685481 T2236 C2466 P50010 oass.SolrIndexSearcher.<init> Opening Searcher@4488d4ca main
[junit4:junit4]   2> 685481 T2250 C2465 P39475 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 685483 T2236 C2466 P50010 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685483 T2202 C2467 P39944 oass.SolrIndexSearcher.<init> Opening Searcher@7a992c6e main
[junit4:junit4]   2> 685484 T2250 C2465 P39475 oass.SolrIndexSearcher.<init> Opening Searcher@3aea2fdd main
[junit4:junit4]   2> 685485 T2242 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4488d4ca main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685485 T2202 C2467 P39944 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685486 T2209 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7a992c6e main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685486 T2260 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3aea2fdd main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 685485 T2216 C2468 P53322 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@60317470 lockFactory=org.apache.lucene.store.NativeFSLockFactory@48e612de),segFN=segments_3,generation=3}
[junit4:junit4]   2> 685487 T2216 C2468 P53322 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 685485 T2250 C2465 P39475 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685487 T2202 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4]   2> 685486 T2236 C2466 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 685489 T2216 C2468 P53322 oass.SolrIndexSearcher.<init> Opening Searcher@2fe11990 main
[junit4:junit4]   2> 685488 T2250 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 685490 T2216 C2468 P53322 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 685491 T2225 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2fe11990 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C2 _1(5.0):C2)}
[junit4:junit4]   2> 685492 T2216 C2468 P53322 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4]   2> 685493 T2252 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4]   2> 685494 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 685498 T2201 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 685501 T2219 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 685504 T2234 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 685506 T2251 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 687508 T2165 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> check const of shard1
[junit4:junit4]   2> client0
[junit4:junit4]   2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:39944_sr","base_url":"http://127.0.0.1:39944/sr","leader":"true"}
[junit4:junit4]   2> 687512 T2202 C2467 P39944 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client1
[junit4:junit4]   2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:53322_sr","base_url":"http://127.0.0.1:53322/sr"}
[junit4:junit4]   2> 687515 T2216 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client2
[junit4:junit4]   2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:50010_sr","base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4]   2> 687518 T2236 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client3
[junit4:junit4]   2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:39475_sr","base_url":"http://127.0.0.1:39475/sr"}
[junit4:junit4]   2> 687521 T2252 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> 687524 T2253 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4]   2> 687530 T2181 C2469 P40343 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={wt=javabin&version=2} {add=[4 (1439069498779893760)]} 0 2
[junit4:junit4]   2> 687536 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:53322/sr/collection1/]
[junit4:junit4]   2> 687537 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:53322/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:true
[junit4:junit4]   2> 687537 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:50010/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4]   2> 687537 T2199 C2467 P39944 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:39475/sr/collection1/ against:[http://127.0.0.1:53322/sr/collection1/] result:false
[junit4:junit4]   2> 687544 T2234 C2466 P50010 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4 (1439069498790379520)]} 0 2
[junit4:junit4]   2> 687544 T2251 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39944/sr/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4 (1439069498790379520)]} 0 2
[junit4:junit4]   2> 687545 T2199 C2467 P39944 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={distrib.from=http://127.0.0.1:39475/sr/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4 (1439069498790379520)]} 0 9
[junit4:junit4]   2> 687546 T2250 C2465 P39475 oasup.LogUpdateProcessor.finish [collection1] webapp=/sr path=/update params={test.distrib.skip.servers=http://127.0.0.1:53322/sr/collection1/&wt=javabin&version=2} {add=[4]} 0 13
[junit4:junit4]   2> 687546 T2165 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 39944
[junit4:junit4]   2> 687569 T2165 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=528298614
[junit4:junit4]   2> 689055 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 689056 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:39944_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4]   2> 689057 T2188 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4]   2> 689064 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 689064 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 689064 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 689064 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 689064 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 689064 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 690572 T2165 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 690573 T2165 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 690575 T2165 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@35d6ca2b
[junit4:junit4]   2> 690580 T2165 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 690582 T2165 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 690582 T2165 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 690583 T2165 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2>  C2467_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:39944_sr, base_url=http://127.0.0.1:39944/sr, leader=true}
[junit4:junit4]   2> 690586 T2165 C2467 P39944 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@673bbff8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2ee94c06),segFN=segments_3,generation=3}
[junit4:junit4]   2> 690587 T2165 C2467 P39944 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 690590 T2165 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 690591 T2165 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4]   2> 690591 T2165 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index;done=false>>]
[junit4:junit4]   2> 690592 T2165 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1/index
[junit4:junit4]   2> 690592 T2165 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1;done=false>>]
[junit4:junit4]   2> 690593 T2165 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372403597233/jetty1
[junit4:junit4]   2> 690596 T2257 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 690596 T2207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 690596 T2187 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 690597 T2240 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 690597 T2207 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 690598 T2165 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/sr,null}
[junit4:junit4]   2> 690597 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 690598 T2194 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 690598 T2207 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 690600 T2194 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 690600 T2223 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 690600 T2257 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 690600 T2187 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 690603 T2223 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4]   2> 690603 T2223 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4]   2> 690604 T2223 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 690604 T2223 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53322/sr/collection1/
[junit4:junit4]   2> 690604 T2223 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr START replicas=[http://127.0.0.1:39944/sr/collection1/, http://127.0.0.1:50010/sr/collection1/, http://127.0.0.1:39475/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 690607 T2223 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:53322/sr  couldn't connect to http://127.0.0.1:39944/sr/collection1/, counting as success
[junit4:junit4]   2> 690608 T2234 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 690608 T2251 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 690609 T2223 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr  Received 6 versions from 127.0.0.1:50010/sr/collection1/
[junit4:junit4]   2> 690609 T2223 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:53322/sr Requesting updates from 127.0.0.1:50010/sr/collection1/n=1 versions=[1439069498790379520]
[junit4:junit4]   2> 690610 T2223 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr  Received 6 versions from 127.0.0.1:39475/sr/collection1/
[junit4:junit4]   2> 690610 T2223 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:53322/sr  Our versions are newer. ourLowThreshold=1439069496422694912 otherHigh=1439069496502386688
[junit4:junit4]   2> 690612 T2233 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={distrib=false&getUpdates=1439069498790379520&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 690615 T2223 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1439069498790379520)]} 0 2
[junit4:junit4]   2> 690615 T2223 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:53322/sr DONE. sync succeeded
[junit4:junit4]   2> 690616 T2223 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 690616 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: try and ask http://127.0.0.1:39944/sr/collection1/ to sync
[junit4:junit4]   2> 690616 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: try and ask http://127.0.0.1:50010/sr/collection1/ to sync
[junit4:junit4]   2> 690617 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: try and ask http://127.0.0.1:39475/sr/collection1/ to sync
[junit4:junit4]   2> 690618 T2223 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:39944/sr/collection1
[junit4:junit4]   2> 690618 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/: Sync failed - asking replica (http://127.0.0.1:39944/sr/collection1/) to recover.
[junit4:junit4]   2> 690619 T2294 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 690619 T2236 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr START replicas=[http://127.0.0.1:53322/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 690620 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr START replicas=[http://127.0.0.1:53322/sr/collection1/] nUpdates=100
[junit4:junit4]   2> 690623 T2216 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 690623 T2219 C2468 P53322 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 690624 T2236 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr  Received 6 versions from 127.0.0.1:53322/sr/collection1/
[junit4:junit4]   2> 690624 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr  Received 6 versions from 127.0.0.1:53322/sr/collection1/
[junit4:junit4]   2> 690624 T2236 C2466 P50010 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:50010/sr  Our versions are newer. ourLowThreshold=1439069496442617856 otherHigh=1439069496502386688
[junit4:junit4]   2> 690625 T2236 C2466 P50010 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:50010/sr DONE. sync succeeded
[junit4:junit4]   2> 690625 T2253 C2465 P39475 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:39475/sr  Our versions are newer. ourLowThreshold=1439069496442617856 otherHigh=1439069496502386688
[junit4:junit4]   2> 690626 T2236 C2466 P50010 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:53322/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=7
[junit4:junit4]   2> 690627 T2253 C2465 P39475 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:39475/sr DONE. sync succeeded
[junit4:junit4]   2> 690627 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/:  sync completed with http://127.0.0.1:50010/sr/collection1/
[junit4:junit4]   2> 690628 T2253 C2465 P39475 oasc.SolrCore.execute [collection1] webapp=/sr path=/get params={sync=http://127.0.0.1:53322/sr/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4]   2> 690629 T2294 oasc.SolrException.log ERROR http://127.0.0.1:53322/sr/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:39944/sr
[junit4:junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]   2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:297)
[junit4:junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:39944 refused
[junit4:junit4]   2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4]   2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4:junit4]   2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4]   2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4]   2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4]   2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4]   2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4]   2> ... 5 more
[junit4:junit4]   2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4]   2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4]   2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4:junit4]   2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4:junit4]   2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4:junit4]   2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4:junit4]   2> at java.net.Socket.connect(Socket.java:579)
[junit4:junit4]   2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4]   2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4]   2> ... 12 more
[junit4:junit4]   2>
[junit4:junit4]   2> 690629 T2223 oasc.SyncStrategy.syncToMe http://127.0.0.1:53322/sr/collection1/:  sync completed with http://127.0.0.1:39475/sr/collection1/
[junit4:junit4]   2> 690630 T2223 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53322/sr/collection1/ shard1
[junit4:junit4]   2> 690630 T2223 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 690636 T2223 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 690638 T2223 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 690650 T2165 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 39944
[junit4:junit4]   2> 692074 T2188 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 692075 T2188 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:39944_sr",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:39944/sr"}
[junit4:junit4]   2> 692090 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live

[...truncated too long message...]

:"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:39475_sr",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:39475/sr"}}}},
[junit4:junit4]   1>           "router":"compositeId"},
[junit4:junit4]   1>         "control_collection":{
[junit4:junit4]   1>           "shards":{"shard1":{
[junit4:junit4]   1>               "range":"80000000-7fffffff",
[junit4:junit4]   1>               "state":"active",
[junit4:junit4]   1>               "replicas":{"core_node1":{
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:40343_sr",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:40343/sr",
[junit4:junit4]   1>                   "leader":"true"}}}},
[junit4:junit4]   1>           "router":"compositeId"}}
[junit4:junit4]   1>   /solr/aliases.json (0)
[junit4:junit4]   1>   /solr/live_nodes (4)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:40343_sr (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:50010_sr (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:39475_sr (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:39944_sr (0)
[junit4:junit4]   1>   /solr/overseer (3)
[junit4:junit4]   1>   DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1>    /solr/overseer/queue (0)
[junit4:junit4]   1>    /solr/overseer/queue-work (0)
[junit4:junit4]   1>    /solr/overseer/collection-queue-work (0)
[junit4:junit4]   1>   /solr/collections (2)
[junit4:junit4]   1>    /solr/collections/collection1 (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/collection1/shards (0)
[junit4:junit4]   1>     /solr/collections/collection1/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89941842148524044-core_node4-n_0000000003 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89941842148524046-core_node1-n_0000000004 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89941842148524042-core_node3-n_0000000002 (0)
[junit4:junit4]   1>     /solr/collections/collection1/leaders (1)
[junit4:junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "core":"collection1",
[junit4:junit4]   1>            "node_name":"127.0.0.1:50010_sr",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:50010/sr"}
[junit4:junit4]   1>    /solr/collections/control_collection (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/control_collection/shards (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/89941842148524035-core_node1-n_0000000000 (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leaders (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "core":"collection1",
[junit4:junit4]   1>            "node_name":"127.0.0.1:40343_sr",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:40343/sr"}
[junit4:junit4]   1>   /solr/overseer_elect (2)
[junit4:junit4]   1>    /solr/overseer_elect/election (4)
[junit4:junit4]   1>     /solr/overseer_elect/election/89941842148524046-127.0.0.1:39944_sr-n_0000000005 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89941842148524044-127.0.0.1:39475_sr-n_0000000004 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89941842148524035-127.0.0.1:40343_sr-n_0000000000 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89941842148524042-127.0.0.1:50010_sr-n_0000000003 (0)
[junit4:junit4]   1>    /solr/overseer_elect/leader (0)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"id":"89941842148524035-127.0.0.1:40343_sr-n_0000000000"}
[junit4:junit4]   1>  /zookeeper (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1>
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=C103F4B8ABD96C0D -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_IE -Dtests.timezone=America/Cordoba -Dtests.file.encoding=UTF-8
[junit4:junit4] FAILURE 99.6s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:39944/sr/collection1lastClient and got 255 from http://127.0.0.1:50010/sr/collection1
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([C103F4B8ABD96C0D:40E57AA0DC860C31]:0)
[junit4:junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
[junit4:junit4]    > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 763848 T2165 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 99606 T2164 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
[junit4:junit4]   2> 763959 T2240 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 763959 T2240 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 763960 T2240 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=DummyCompressingStoredFields(storedFieldsFormat=CompressingStoredFieldsFormat(compressionMode=DUMMY, chunkSize=500), termVectorsFormat=CompressingTermVectorsFormat(compressionMode=DUMMY, chunkSize=500)), sim=DefaultSimilarity, locale=en_IE, timezone=America/Cordoba
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=3,free=137002440,total=312283136
[junit4:junit4]   2> NOTE: All tests run in this JVM: [PeerSyncTest, BasicFunctionalityTest, TestDynamicFieldResource, FastVectorHighlighterTest, TestFieldTypeResource, JsonLoaderTest, TestFunctionQuery, AnalysisAfterCoreReloadTest, TestLMDirichletSimilarityFactory, MinimalSchemaTest, DirectUpdateHandlerOptimizeTest, BasicDistributedZkTest, UUIDFieldTest, TestCopyFieldCollectionResource, TestSolrCoreProperties, TestTrie, OverseerCollectionProcessorTest, TestPhraseSuggestions, TestDefaultSearchFieldResource, TestDocSet, ClusterStateUpdateTest, LeaderElectionTest, RAMDirectoryFactoryTest, TestPseudoReturnFields, ShowFileRequestHandlerTest, TermVectorComponentTest, MoreLikeThisHandlerTest, DistributedQueryElevationComponentTest, TestLFUCache, TestManagedSchemaFieldResource, TestFastWriter, BadCopyFieldTest, TestSolrIndexConfig, DistributedTermsComponentTest, TestSolrDeletionPolicy2, SimpleFacetsTest, TestPluginEnable, TestValueSourceCache, DocValuesTest, TestStressRecovery, RecoveryZkTest, DirectSolrConnectionTest, TestQueryUtils, QueryElevationComponentTest, TestPropInjectDefaults, TestStressVersions, TestWriterPerf, SliceStateTest, TestGroupingSearch, TestFastLRUCache, SpellingQueryConverterTest, TestReversedWildcardFilterFactory, BlockDirectoryTest, BadComponentTest, CircularListTest, TestBadConfig, TestFieldTypeCollectionResource, IndexSchemaTest, UnloadDistributedZkTest, QueryParsingTest, TestPerFieldSimilarity, SpatialFilterTest, AliasIntegrationTest, TestSearchPerf, TestJoin, PreAnalyzedUpdateProcessorTest, SchemaVersionSpecificBehaviorTest, ParsingFieldUpdateProcessorsTest, SynonymTokenizerTest, TestSolrQueryParserDefaultOperatorResource, ScriptEngineTest, WordBreakSolrSpellCheckerTest, SuggesterFSTTest, TestQuerySenderListener, TestDocumentBuilder, BasicDistributedZk2Test, SyncSliceTest]
[junit4:junit4] Completed on J0 in 99.88s, 1 test, 1 failure <<< FAILURES!

[...truncated 485 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 1 failure, 20 ignored (13 assumptions)

Total time: 39 minutes 35 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build # 6347 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6347/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:43365/collection1lastClient and got 256 from http://127.0.0.1:55874/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:43365/collection1lastClient and got 256 from http://127.0.0.1:55874/collection1
        at __randomizedtesting.SeedInfo.seed([EE579D43B894FBE1:6FB1135BCFCB9BDD]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9219 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 31314 T3 oasu.ConcurrentLRUCache.finalize ERROR ConcurrentLRUCache was not destroyed prior to finalize(), indicates a bug -- POSSIBLE RESOURCE LEAK!!!
[junit4:junit4]   2> 31317 T161 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
[junit4:junit4]   2> 31323 T161 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372408195692
[junit4:junit4]   2> 31326 T161 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 31329 T162 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 31429 T161 oasc.ZkTestServer.run start zk server on port:51822
[junit4:junit4]   2> 34479 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 34512 T168 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@664c6b6c name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 34513 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 34514 T161 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 34538 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 34540 T170 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5bc1d381 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 34540 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 34542 T161 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 34546 T161 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 34550 T161 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 34552 T161 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 34556 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 34556 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 34562 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 34562 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 34567 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 34567 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 34571 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 34572 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 34575 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 34576 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 34580 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 34580 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 34584 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 34584 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 34588 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 34589 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 34593 T161 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 34593 T161 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 34667 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 34671 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:60204
[junit4:junit4]   2> 34672 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 34672 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 34673 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967
[junit4:junit4]   2> 34673 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/solr.xml
[junit4:junit4]   2> 34673 T161 oasc.CoreContainer.<init> New CoreContainer 720319641
[junit4:junit4]   2> 34674 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/'
[junit4:junit4]   2> 34674 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/'
[junit4:junit4]   2> 34725 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 34725 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 34726 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 34726 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 34726 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 34727 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 34728 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 34732 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 34732 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4]   2> 34740 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 34741 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 34744 T181 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2fcc2471 name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 34744 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 34748 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 34753 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 34755 T183 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e8d95d name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 34755 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 34760 T161 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 34767 T161 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 34775 T161 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 34778 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:60204_
[junit4:junit4]   2> 34781 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:60204_
[junit4:junit4]   2> 34785 T161 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 34795 T161 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 34799 T161 oasc.Overseer.start Overseer (id=89942143517917187-127.0.0.1:60204_-n_0000000000) starting
[junit4:junit4]   2> 34807 T161 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 34817 T185 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 34817 T161 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 34820 T161 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 34823 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 34830 T184 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 34834 T186 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 34834 T186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 34836 T186 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 36337 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 36339 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:60204_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:60204"}
[junit4:junit4]   2> 36339 T184 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 36341 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 36351 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 36837 T186 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 36837 T186 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1
[junit4:junit4]   2> 36837 T186 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 36839 T186 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 36839 T186 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 36841 T186 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/'
[junit4:junit4]   2> 36842 T186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/lib/README' to classloader
[junit4:junit4]   2> 36843 T186 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 36886 T186 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 36917 T186 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 36919 T186 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 36925 T186 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 37310 T186 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 37317 T186 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 37319 T186 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 37341 T186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 37345 T186 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 37350 T186 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 37352 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 37352 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 37352 T186 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 37354 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 37354 T186 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 37354 T186 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 37355 T186 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372408198967/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/
[junit4:junit4]   2> 37355 T186 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4]   2> 37356 T186 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data
[junit4:junit4]   2> 37356 T186 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/index/
[junit4:junit4]   2> 37356 T186 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 37356 T186 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/control/data/index
[junit4:junit4]   2> 37358 T186 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_1,generation=1}
[junit4:junit4]   2> 37358 T186 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 37360 T186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 37361 T186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 37362 T186 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 37362 T186 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 37363 T186 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 37363 T186 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 37363 T186 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 37364 T186 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 37364 T186 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 37366 T186 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 37369 T186 oass.SolrIndexSearcher.<init> Opening Searcher@62360ab1 main
[junit4:junit4]   2> 37370 T186 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 37370 T186 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 40380 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@62360ab1 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 40383 T186 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 40384 T186 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:60204 collection:control_collection shard:shard1
[junit4:junit4]   2> 40390 T186 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=30000&socketTimeout=30000&retry=false
[junit4:junit4]   2> 40396 T186 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 40410 T186 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 40416 T186 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 40417 T186 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 40419 T186 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:60204/collection1/
[junit4:junit4]   2> 40420 T186 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 40420 T186 oasc.SyncStrategy.syncToMe http://127.0.0.1:60204/collection1/ has no replicas
[junit4:junit4]   2> 40421 T186 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:60204/collection1/ shard1
[junit4:junit4]   2> 40421 T186 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 40863 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 40876 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 40890 T186 oasc.ZkController.register We are http://127.0.0.1:60204/collection1/ and leader is http://127.0.0.1:60204/collection1/
[junit4:junit4]   2> 40890 T186 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:60204
[junit4:junit4]   2> 40890 T186 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 40890 T186 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 40891 T186 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 40893 T186 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 40894 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 40895 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 40895 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 40899 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 40902 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 40903 T190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@42408073 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 40904 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 40906 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 40911 T161 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 40991 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 40993 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43365
[junit4:junit4]   2> 40994 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 40994 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 40995 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280
[junit4:junit4]   2> 40995 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/solr.xml
[junit4:junit4]   2> 40995 T161 oasc.CoreContainer.<init> New CoreContainer 1430281418
[junit4:junit4]   2> 40996 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/'
[junit4:junit4]   2> 40996 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/'
[junit4:junit4]   2> 41046 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 41046 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 41047 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 41048 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 41048 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 41049 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 41049 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 41053 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 41054 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4]   2> 41054 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 41055 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 41057 T201 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4e5acdf name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 41057 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 41063 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 41068 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 41070 T203 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94c91f3 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 41070 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 41075 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 42080 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43365_
[junit4:junit4]   2> 42081 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43365_
[junit4:junit4]   2> 42085 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 42086 T190 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 42086 T203 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 42088 T183 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 42095 T204 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 42095 T204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 42096 T204 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 42383 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 42385 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:60204_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:60204"}
[junit4:junit4]   2> 42390 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:43365_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:43365"}
[junit4:junit4]   2> 42390 T184 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 42390 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 42397 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 42397 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 42397 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 43097 T204 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 43097 T204 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1
[junit4:junit4]   2> 43098 T204 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 43098 T204 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 43099 T204 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 43100 T204 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/'
[junit4:junit4]   2> 43101 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/lib/README' to classloader
[junit4:junit4]   2> 43101 T204 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 43133 T204 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 43164 T204 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 43166 T204 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 43174 T204 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 43550 T204 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 43557 T204 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 43559 T204 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 43585 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 43590 T204 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 43600 T204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 43602 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 43603 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 43603 T204 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 43605 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 43605 T204 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 43606 T204 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 43606 T204 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372408205280/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/
[junit4:junit4]   2> 43607 T204 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4]   2> 43608 T204 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1
[junit4:junit4]   2> 43609 T204 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index/
[junit4:junit4]   2> 43609 T204 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 43609 T204 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index
[junit4:junit4]   2> 43612 T204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 43612 T204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 43615 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 43616 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 43617 T204 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 43618 T204 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 43619 T204 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 43619 T204 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 43620 T204 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 43621 T204 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 43622 T204 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 43626 T204 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 43629 T204 oass.SolrIndexSearcher.<init> Opening Searcher@2c81a8a main
[junit4:junit4]   2> 43631 T204 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 43631 T204 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 43636 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2c81a8a main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 43638 T204 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 43639 T204 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:43365 collection:collection1 shard:shard1
[junit4:junit4]   2> 43640 T204 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 43648 T204 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 43651 T204 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 43651 T204 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 43651 T204 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43365/collection1/
[junit4:junit4]   2> 43651 T204 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 43651 T204 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/ has no replicas
[junit4:junit4]   2> 43652 T204 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:43365/collection1/ shard1
[junit4:junit4]   2> 43652 T204 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 43905 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 43921 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 43921 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 43921 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 43959 T204 oasc.ZkController.register We are http://127.0.0.1:43365/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4]   2> 43960 T204 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:43365
[junit4:junit4]   2> 43960 T204 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 43961 T204 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 43961 T204 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 43965 T204 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 43968 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 43968 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 43969 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 44055 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 44057 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58508
[junit4:junit4]   2> 44058 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 44058 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 44059 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347
[junit4:junit4]   2> 44059 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/solr.xml
[junit4:junit4]   2> 44059 T161 oasc.CoreContainer.<init> New CoreContainer 408463892
[junit4:junit4]   2> 44060 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/'
[junit4:junit4]   2> 44060 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/'
[junit4:junit4]   2> 44114 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 44115 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 44115 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 44116 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 44116 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 44117 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 44117 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 44118 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 44118 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 44119 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 44125 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 44125 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4]   2> 44126 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 44127 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 44130 T217 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@30783ead name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 44131 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 44134 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 44141 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 44144 T219 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2bdd746b name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 44145 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 44155 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 45163 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58508_
[junit4:junit4]   2> 45164 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58508_
[junit4:junit4]   2> 45167 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 45167 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 45167 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 45169 T219 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 45171 T190 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 45171 T183 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 45171 T203 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 45179 T220 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 45180 T220 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 45181 T220 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 45433 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 45434 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:43365_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:43365"}
[junit4:junit4]   2> 45439 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:58508_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58508"}
[junit4:junit4]   2> 45439 T184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 45440 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 45445 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 45445 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 45445 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 45446 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 46181 T220 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 46182 T220 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1
[junit4:junit4]   2> 46183 T220 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 46184 T220 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 46185 T220 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 46188 T220 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/'
[junit4:junit4]   2> 46190 T220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/lib/README' to classloader
[junit4:junit4]   2> 46191 T220 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 46238 T220 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 46275 T220 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 46277 T220 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 46281 T220 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 46660 T220 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 46668 T220 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 46671 T220 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 46696 T220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 46701 T220 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 46705 T220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 46707 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 46707 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 46708 T220 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 46709 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 46709 T220 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 46710 T220 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 46710 T220 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372408208347/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/
[junit4:junit4]   2> 46710 T220 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4]   2> 46711 T220 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2
[junit4:junit4]   2> 46711 T220 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/index/
[junit4:junit4]   2> 46711 T220 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 46712 T220 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty2/index
[junit4:junit4]   2> 46714 T220 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_1,generation=1}
[junit4:junit4]   2> 46714 T220 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 46717 T220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 46717 T220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 46718 T220 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 46718 T220 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 46719 T220 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 46719 T220 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 46720 T220 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 46720 T220 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 46721 T220 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 46724 T220 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 46727 T220 oass.SolrIndexSearcher.<init> Opening Searcher@7b52d3fe main
[junit4:junit4]   2> 46727 T220 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 46728 T220 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 46736 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7b52d3fe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 46740 T220 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 46741 T220 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:58508 collection:collection1 shard:shard1
[junit4:junit4]   2> 46749 T220 oasc.ZkController.register We are http://127.0.0.1:58508/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4]   2> 46749 T220 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:58508
[junit4:junit4]   2> 46749 T220 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 46750 T220 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C23 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C23_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4]   2> 46754 T222 C23 P58508 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 46755 T222 C23 P58508 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 46755 T220 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 46756 T222 C23 P58508 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 46757 T222 C23 P58508 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 46758 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 46759 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 46760 T222 C23 P58508 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 46760 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 46778 T197 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 46854 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 46857 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55874
[junit4:junit4]   2> 46857 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 46858 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 46858 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138
[junit4:junit4]   2> 46858 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/solr.xml
[junit4:junit4]   2> 46859 T161 oasc.CoreContainer.<init> New CoreContainer 383111009
[junit4:junit4]   2> 46859 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/'
[junit4:junit4]   2> 46859 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/'
[junit4:junit4]   2> 46913 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 46914 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 46914 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 46915 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 46915 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 46915 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 46916 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 46916 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 46917 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 46917 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 46922 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 46923 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4]   2> 46923 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 46924 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 46926 T234 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1a41c343 name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 46927 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 46929 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 46935 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 46936 T236 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5cc01173 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 46937 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 46942 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 46960 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 46963 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:58508_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58508"}
[junit4:junit4]   2> 46975 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 46976 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 46976 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 46976 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 46976 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 47779 T197 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 47780 T197 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:58508_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1005
[junit4:junit4]   2> 47946 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55874_
[junit4:junit4]   2> 47949 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55874_
[junit4:junit4]   2> 47954 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 47954 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 47954 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 47954 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 47956 T236 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 47957 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 47961 T190 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 47963 T183 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 47963 T203 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 47963 T219 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 47971 T237 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 47971 T237 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 47974 T237 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 48488 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 48489 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:55874_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:55874"}
[junit4:junit4]   2> 48489 T184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 48489 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 48494 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 48494 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 48494 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 48495 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 48495 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 48975 T237 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 48976 T237 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1
[junit4:junit4]   2> 48976 T237 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 48978 T237 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 48978 T237 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 48982 T237 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/'
[junit4:junit4]   2> 48983 T237 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/lib/README' to classloader
[junit4:junit4]   2> 48983 T237 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 49033 T237 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 49081 T237 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 49084 T237 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 49088 T237 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 49435 T237 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 49441 T237 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 49444 T237 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 49465 T237 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 49468 T237 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 49472 T237 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 49473 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 49474 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 49474 T237 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 49475 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 49476 T237 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 49476 T237 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 49476 T237 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372408211138/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/
[junit4:junit4]   2> 49476 T237 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4]   2> 49477 T237 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3
[junit4:junit4]   2> 49477 T237 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/index/
[junit4:junit4]   2> 49478 T237 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 49478 T237 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty3/index
[junit4:junit4]   2> 49479 T237 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_1,generation=1}
[junit4:junit4]   2> 49480 T237 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 49482 T237 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 49482 T237 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 49483 T237 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 49483 T237 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 49484 T237 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 49484 T237 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 49485 T237 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 49485 T237 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 49486 T237 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 49488 T237 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 49490 T237 oass.SolrIndexSearcher.<init> Opening Searcher@ffe13fe main
[junit4:junit4]   2> 49491 T237 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 49491 T237 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 49496 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@ffe13fe main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 49498 T237 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 49498 T237 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:55874 collection:collection1 shard:shard1
[junit4:junit4]   2> 49503 T237 oasc.ZkController.register We are http://127.0.0.1:55874/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4]   2> 49503 T237 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:55874
[junit4:junit4]   2> 49503 T237 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 49504 T237 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C24_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4]   2> 49504 T239 C24 P55874 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 49505 T239 C24 P55874 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 49505 T237 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 49505 T239 C24 P55874 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 49506 T239 C24 P55874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 49507 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 49507 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 49508 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 49508 T239 C24 P55874 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 49513 T196 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 49578 T161 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 49581 T161 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57393
[junit4:junit4]   2> 49582 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 49583 T161 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 49583 T161 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881
[junit4:junit4]   2> 49583 T161 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/solr.xml
[junit4:junit4]   2> 49583 T161 oasc.CoreContainer.<init> New CoreContainer 1726007250
[junit4:junit4]   2> 49584 T161 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/'
[junit4:junit4]   2> 49584 T161 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/'
[junit4:junit4]   2> 49634 T161 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 49634 T161 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 49634 T161 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 49635 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 49635 T161 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 49635 T161 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 49636 T161 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 49636 T161 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 49636 T161 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 49637 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 49640 T161 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 49641 T161 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51822/solr
[junit4:junit4]   2> 49641 T161 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 49642 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 49644 T251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d2703fb name:ZooKeeperConnection Watcher:127.0.0.1:51822 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 49645 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 49647 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 49651 T161 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 49653 T253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@305b8b04 name:ZooKeeperConnection Watcher:127.0.0.1:51822/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 49653 T161 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 49658 T161 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C25_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4]   2> 49782 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43365/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 49783 T222 C25 P58508 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4]   2> 49784 T222 C25 P58508 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 49784 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 49784 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 49785 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 49785 T222 C25 P58508 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 49786 T222 C25 P58508 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43365/collection1/. core=collection1
[junit4:junit4]   2> 49786 T222 C25 P58508 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C26 name=collection1 org.apache.solr.core.SolrCore@3e59c0b url=http://127.0.0.1:43365/collection1 node=127.0.0.1:43365_ C26_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4]   2> 49789 T198 C26 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 49804 T197 C26 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 49806 T197 C26 P43365 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4]   2> 49807 T197 C26 P43365 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 49807 T197 C26 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 49808 T197 C26 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@7b7767ee realtime
[junit4:junit4]   2> 49808 T197 C26 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 49809 T197 C26 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 15
[junit4:junit4]   2> 49832 T222 C25 P58508 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 49833 T222 C25 P58508 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4]   2> 49840 T222 C25 P58508 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 49845 T199 C26 P43365 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 49845 T199 C26 P43365 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4]   2> 49846 T222 C25 P58508 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 49847 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 49847 T222 C25 P58508 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 49847 T222 C25 P58508 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 49849 T222 C25 P58508 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 50001 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 50002 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:55874_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:55874"}
[junit4:junit4]   2> 50007 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:58508_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:58508"}
[junit4:junit4]   2> 50013 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50013 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50013 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50013 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50013 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50014 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50514 T196 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 50515 T196 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:55874_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4]   2> 50662 T161 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57393_
[junit4:junit4]   2> 50663 T161 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57393_
[junit4:junit4]   2> 50666 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50667 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50667 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50666 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50666 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 50667 T236 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 50668 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 50671 T183 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 50672 T190 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 50672 T219 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 50672 T203 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 50672 T253 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 50676 T255 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 50677 T255 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 50678 T255 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 51527 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 51528 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:57393_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:57393"}
[junit4:junit4]   2> 51528 T184 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 51529 T184 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 51534 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 51534 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 51534 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 51534 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 51534 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 51535 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 51679 T255 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 51679 T255 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1
[junit4:junit4]   2> 51679 T255 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 51681 T255 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 51681 T255 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 51683 T255 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/'
[junit4:junit4]   2> 51684 T255 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/lib/README' to classloader
[junit4:junit4]   2> 51685 T255 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 51724 T255 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 51754 T255 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 51756 T255 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 51773 T255 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 52137 T255 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 52143 T255 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 52145 T255 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 52182 T255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 52190 T255 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 52195 T255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 52197 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 52198 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 52198 T255 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 52199 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 52200 T255 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 52200 T255 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 52201 T255 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372408213881/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/
[junit4:junit4]   2> 52201 T255 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@522103ff
[junit4:junit4]   2> 52202 T255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4
[junit4:junit4]   2> 52202 T255 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/index/
[junit4:junit4]   2> 52203 T255 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 52203 T255 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty4/index
[junit4:junit4]   2> 52205 T255 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_1,generation=1}
[junit4:junit4]   2> 52206 T255 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 52209 T255 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 52209 T255 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 52210 T255 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 52211 T255 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 52212 T255 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 52212 T255 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 52213 T255 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 52214 T255 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 52214 T255 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 52218 T255 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 52221 T255 oass.SolrIndexSearcher.<init> Opening Searcher@e2471c7 main
[junit4:junit4]   2> 52222 T255 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 52223 T255 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 52230 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@e2471c7 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 52233 T255 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 52234 T255 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:57393 collection:collection1 shard:shard1
[junit4:junit4]   2> 52238 T255 oasc.ZkController.register We are http://127.0.0.1:57393/collection1/ and leader is http://127.0.0.1:43365/collection1/
[junit4:junit4]   2> 52238 T255 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:57393
[junit4:junit4]   2> 52238 T255 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 52239 T255 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C27 name=collection1 org.apache.solr.core.SolrCore@6c18b5da url=http://127.0.0.1:57393/collection1 node=127.0.0.1:57393_ C27_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4]   2> 52239 T257 C27 P57393 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 52240 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 52240 T255 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 52240 T257 C27 P57393 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 52240 T257 C27 P57393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 52241 T161 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 52242 T161 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 52242 T257 C27 P57393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 52242 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 52246 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 52248 T199 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 52249 T161 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4]   2> 52249 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4]   2> 52250 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C28_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4]   2> 52516 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43365/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 52516 T239 C28 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4]   2> 52517 T239 C28 P55874 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 52518 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 52518 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 52519 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 52519 T239 C28 P55874 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 52520 T239 C28 P55874 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43365/collection1/. core=collection1
[junit4:junit4]   2> 52520 T239 C28 P55874 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@3e59c0b url=http://127.0.0.1:43365/collection1 node=127.0.0.1:43365_ C29_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4]   2> 52521 T197 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 52531 T196 C29 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 52532 T196 C29 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 52533 T196 C29 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 52533 T196 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> 52535 T239 C28 P55874 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 52535 T239 C28 P55874 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 52540 T197 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4]   2> 52541 T239 C28 P55874 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 52542 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 52542 T239 C28 P55874 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 52542 T239 C28 P55874 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 52545 T239 C28 P55874 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 53041 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 53042 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:57393_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:57393"}
[junit4:junit4]   2> 53047 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:55874_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:55874"}
[junit4:junit4]   2> 53054 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 53054 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 53054 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 53054 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 53054 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 53055 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 53248 T199 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 53249 T199 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:57393_&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 53252 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 54255 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2>  C27_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4]   2> 55250 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:43365/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 55250 T257 C27 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4]   2> 55251 T257 C27 P57393 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 55251 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 55251 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 55251 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 55251 T257 C27 P57393 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 55252 T257 C27 P57393 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:43365/collection1/. core=collection1
[junit4:junit4]   2> 55252 T257 C27 P57393 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 55253 T199 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 55257 T197 C29 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 55258 T197 C29 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 55258 T197 C29 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 55258 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 55258 T197 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4]   2> 55260 T257 C27 P57393 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 55260 T257 C27 P57393 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 55262 T199 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 55263 T257 C27 P57393 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 55263 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 55263 T257 C27 P57393 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 55264 T257 C27 P57393 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 55265 T257 C27 P57393 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 56069 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 56071 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:57393_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:57393"}
[junit4:junit4]   2> 56079 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 56080 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 56080 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 56080 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 56080 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 56080 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 56261 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 56263 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@8635d12 url=http://127.0.0.1:60204/collection1 node=127.0.0.1:60204_ C30_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60204_, base_url=http://127.0.0.1:60204, leader=true}
[junit4:junit4]   2> 56273 T178 C30 P60204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 56275 T178 C30 P60204 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_1,generation=1}
[junit4:junit4]   2> 56276 T178 C30 P60204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 56276 T178 C30 P60204 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 56276 T178 C30 P60204 oass.SolrIndexSearcher.<init> Opening Searcher@3eb78a85 main
[junit4:junit4]   2> 56277 T178 C30 P60204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 56278 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3eb78a85 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 56278 T178 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2>  C27_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4]   2> 56298 T249 C27 P57393 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43365/collection1/, StdNode: http://127.0.0.1:58508/collection1/, StdNode: http://127.0.0.1:55874/collection1/, StdNode: http://127.0.0.1:57393/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 56307 T246 C27 P57393 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 56307 T198 C29 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 56308 T198 C29 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 56309 T198 C29 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@2ca08829 main
[junit4:junit4]   2> 56309 T246 C27 P57393 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_1,generation=1}
[junit4:junit4]   2> 56310 T198 C29 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 56311 T246 C27 P57393 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 56312 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2ca08829 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 56312 T246 C27 P57393 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 56314 T198 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> ASYNC  NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C31_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4]   2> 56314 T214 C31 P58508 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 56314 T246 C27 P57393 oass.SolrIndexSearcher.<init> Opening Searcher@4a3ece73 main
[junit4:junit4]   2>  C28_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4]   2> 56315 T232 C28 P55874 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 56316 T246 C27 P57393 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 56317 T214 C31 P58508 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_1,generation=1}
[junit4:junit4]   2> 56318 T214 C31 P58508 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 56318 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4a3ece73 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 56319 T232 C28 P55874 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_1,generation=1}
[junit4:junit4]   2> 56320 T246 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4]   2> 56319 T214 C31 P58508 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 56321 T232 C28 P55874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 56322 T214 C31 P58508 oass.SolrIndexSearcher.<init> Opening Searcher@a873204 main
[junit4:junit4]   2> 56323 T232 C28 P55874 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 56323 T214 C31 P58508 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 56324 T232 C28 P55874 oass.SolrIndexSearcher.<init> Opening Searcher@5a2e8b7e main
[junit4:junit4]   2> 56325 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@a873204 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 56325 T232 C28 P55874 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 56326 T214 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4]   2> 56327 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a2e8b7e main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 56328 T232 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4]   2> 56330 T249 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4]   2> 56332 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 56338 T197 C29 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4]   2> 56343 T215 C31 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 56348 T230 C28 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 56353 T248 C27 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 58360 T179 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439074324554186752)} 0 3
[junit4:junit4]   2> 58371 T231 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439074324562575360&update.from=http://127.0.0.1:43365/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 2
[junit4:junit4]   2> 58371 T212 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439074324562575360&update.from=http://127.0.0.1:43365/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 1
[junit4:junit4]   2> 58371 T248 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=FROMLEADER&_version_=-1439074324562575360&update.from=http://127.0.0.1:43365/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 1
[junit4:junit4]   2> 58372 T197 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439074324562575360)} 0 7
[junit4:junit4]   2> 58373 T229 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 11
[junit4:junit4]   2> 58378 T178 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0 (1439074324573061120)]} 0 3
[junit4:junit4]   2> 58395 T249 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 3
[junit4:junit4]   2> 58395 T232 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 3
[junit4:junit4]   2> 58396 T213 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 4
[junit4:junit4]   2> 58397 T196 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:57393/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439074324584595456)]} 0 11
[junit4:junit4]   2> 58398 T246 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[0]} 0 17
[junit4:junit4]   2> 58403 T177 C30 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1439074324599275520)]} 0 3
[junit4:junit4]   2> 58416 T230 C28 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 2
[junit4:junit4]   2> 58418 T214 C31 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 4
[junit4:junit4]   2> 58417 T247 C27 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 3
[junit4:junit4]   2> 58419 T199 C29 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[1 (1439074324603469824)]} 0 14
[junit4:junit4]   2> ASYNC  NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@8635d12 url=http://127.0.0.1:60204/collection1 node=127.0.0.1:60204_ C32_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:60204_, base_url=http://127.0.0.1:60204, leader=true}
[junit4:junit4]   2> 58433 T176 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[2 (1439074324631781376)]} 0 2
[junit4:junit4]   2> ASYNC  NEW_CORE C33 name=collection1 org.apache.solr.core.SolrCore@3e59c0b url=http://127.0.0.1:43365/collection1 node=127.0.0.1:43365_ C33_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4]   2> 58440 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:58508/collection1/]
[junit4:junit4]   2> 58440 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:58508/collection1/ against:[http://127.0.0.1:58508/collection1/] result:true
[junit4:junit4]   2> 58440 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:55874/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4]   2> 58441 T198 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:57393/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4]   2> ASYNC  NEW_CORE C34 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C34_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4]   2> 58447 T231 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2 (1439074324641218560)]} 0 2
[junit4:junit4]   2> ASYNC  NEW_CORE C35 name=collection1 org.apache.solr.core.SolrCore@6c18b5da url=http://127.0.0.1:57393/collection1 node=127.0.0.1:57393_ C35_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4]   2> 58447 T249 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2 (1439074324641218560)]} 0 2
[junit4:junit4]   2> 58448 T198 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:57393/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2 (1439074324641218560)]} 0 9
[junit4:junit4]   2> 58449 T248 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[2]} 0 13
[junit4:junit4]   2> 58455 T179 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[3 (1439074324652752896)]} 0 3
[junit4:junit4]   2> 58458 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/]
[junit4:junit4]   2> 58459 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:58508/collection1/ against:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/] result:true
[junit4:junit4]   2> 58459 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:55874/collection1/ against:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/] result:true
[junit4:junit4]   2> 58459 T197 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:57393/collection1/ against:[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/] result:false
[junit4:junit4]   2> 58466 T246 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[3 (1439074324661141504)]} 0 2
[junit4:junit4]   2> 58467 T197 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&test.distrib.skip.servers=http://127.0.0.1:55874/collection1/&wt=javabin&version=2} {add=[3 (1439074324661141504)]} 0 9
[junit4:junit4]   2> 58470 T178 C32 P60204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58479 T178 C32 P60204 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5324396a lockFactory=org.apache.lucene.store.NativeFSLockFactory@59e05bf9),segFN=segments_2,generation=2}
[junit4:junit4]   2> 58480 T178 C32 P60204 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 58487 T178 C32 P60204 oass.SolrIndexSearcher.<init> Opening Searcher@3043b1cc main
[junit4:junit4]   2> 58488 T178 C32 P60204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58489 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3043b1cc main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4]   2> 58490 T178 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 21
[junit4:junit4]   2> 58493 T229 C34 P55874 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43365/collection1/, StdNode: http://127.0.0.1:58508/collection1/, StdNode: http://127.0.0.1:55874/collection1/, StdNode: http://127.0.0.1:57393/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> ASYNC  NEW_CORE C36 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C36_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4]   2> 58496 T215 C36 P58508 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58497 T247 C35 P57393 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58497 T232 C34 P55874 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58497 T196 C33 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58505 T215 C36 P58508 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_2,generation=2}
[junit4:junit4]   2> 58506 T215 C36 P58508 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 58507 T232 C34 P55874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_2,generation=2}
[junit4:junit4]   2> 58508 T232 C34 P55874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 58507 T247 C35 P57393 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@30f0e803 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7736970f),segFN=segments_2,generation=2}
[junit4:junit4]   2> 58509 T247 C35 P57393 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 58510 T215 C36 P58508 oass.SolrIndexSearcher.<init> Opening Searcher@42ccfd11 main
[junit4:junit4]   2> 58511 T215 C36 P58508 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58511 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@42ccfd11 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c2)}
[junit4:junit4]   2> 58512 T247 C35 P57393 oass.SolrIndexSearcher.<init> Opening Searcher@4e383c54 main
[junit4:junit4]   2> 58512 T215 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 16
[junit4:junit4]   2> 58512 T247 C35 P57393 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58513 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4e383c54 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4]   2> 58514 T232 C34 P55874 oass.SolrIndexSearcher.<init> Opening Searcher@7ca99a28 main
[junit4:junit4]   2> 58514 T247 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 18
[junit4:junit4]   2> 58514 T232 C34 P55874 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58515 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7ca99a28 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c3)}
[junit4:junit4]   2> 58516 T232 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 20
[junit4:junit4]   2> 58519 T196 C33 P43365 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_2,generation=2}
[junit4:junit4]   2> 58519 T196 C33 P43365 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 58523 T196 C33 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@7c7686f8 main
[junit4:junit4]   2> 58523 T196 C33 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58524 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7c7686f8 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4]   2> 58525 T196 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 29
[junit4:junit4]   2> 58526 T229 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 33
[junit4:junit4]   2> 58526 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4]   2> 58527 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 58528 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 58531 T199 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 58534 T212 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4]   2> 58538 T230 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=1
[junit4:junit4]   2> 58541 T249 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 58543 T161 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 58549 T231 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 58550 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 58555 T198 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 58556 T198 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:43365/collection1/
[junit4:junit4]   2> 58556 T198 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:43365 START replicas=[http://127.0.0.1:58508/collection1/, http://127.0.0.1:55874/collection1/, http://127.0.0.1:57393/collection1/] nUpdates=100
[junit4:junit4]   2> 58560 T229 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 58560 T246 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 58560 T213 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 58560 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365  Received 4 versions from 127.0.0.1:55874/collection1/
[junit4:junit4]   2> 58561 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365  Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324641218560
[junit4:junit4]   2> 58561 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365  Received 5 versions from 127.0.0.1:57393/collection1/
[junit4:junit4]   2> 58562 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365  Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324641218560
[junit4:junit4]   2> 58562 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365  Received 3 versions from 127.0.0.1:58508/collection1/
[junit4:junit4]   2> 58562 T198 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:43365  Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324603469824
[junit4:junit4]   2> 58563 T198 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:43365 DONE. sync succeeded
[junit4:junit4]   2> 58563 T198 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 58563 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: try and ask http://127.0.0.1:58508/collection1/ to sync
[junit4:junit4]   2> 58564 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: try and ask http://127.0.0.1:55874/collection1/ to sync
[junit4:junit4]   2> 58564 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/: try and ask http://127.0.0.1:57393/collection1/ to sync
[junit4:junit4]   2> 58566 T214 C36 P58508 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4]   2> 58567 T249 C35 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4]   2> 58568 T232 C34 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 START replicas=[http://127.0.0.1:43365/collection1/] nUpdates=100
[junit4:junit4]   2> 58570 T199 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 58571 T196 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 58571 T214 C36 P58508 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508  Received 5 versions from 127.0.0.1:43365/collection1/
[junit4:junit4]   2> 58572 T249 C35 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393  Received 5 versions from 127.0.0.1:43365/collection1/
[junit4:junit4]   2> 58571 T197 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 58572 T249 C35 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393  Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324641218560
[junit4:junit4]   2> 58572 T214 C36 P58508 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:58508 Requesting updates from 127.0.0.1:43365/collection1/n=2 versions=[1439074324661141504, 1439074324641218560]
[junit4:junit4]   2> 58573 T232 C34 P55874 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:55874  Received 5 versions from 127.0.0.1:43365/collection1/
[junit4:junit4]   2> 58573 T249 C35 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 DONE. sync succeeded
[junit4:junit4]   2> 58574 T232 C34 P55874 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:55874 Requesting updates from 127.0.0.1:43365/collection1/n=1 versions=[1439074324661141504]
[junit4:junit4]   2> 58575 T249 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:43365/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=8
[junit4:junit4]   2> 58576 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/:  sync completed with http://127.0.0.1:57393/collection1/
[junit4:junit4]   2> 58576 T199 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439074324661141504,1439074324641218560&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 58578 T196 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439074324661141504&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 58583 T232 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] {add=[3 (1439074324661141504)]} 0 4
[junit4:junit4]   2> 58583 T232 C34 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 DONE. sync succeeded
[junit4:junit4]   2> 58584 T232 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:43365/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=16
[junit4:junit4]   2> 58584 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/:  sync completed with http://127.0.0.1:55874/collection1/
[junit4:junit4]   2> 58586 T214 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439074324641218560), 3 (1439074324661141504)]} 0 9
[junit4:junit4]   2> 58586 T214 C36 P58508 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 DONE. sync succeeded
[junit4:junit4]   2> 58587 T214 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:43365/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=20
[junit4:junit4]   2> 58587 T198 oasc.SyncStrategy.syncToMe http://127.0.0.1:43365/collection1/:  sync completed with http://127.0.0.1:58508/collection1/
[junit4:junit4]   2> 58588 T198 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=33
[junit4:junit4]   2> 58588 T231 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=40
[junit4:junit4]   2> 58589 T161 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 58589 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 58590 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 58591 T161 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 58594 T177 C32 P60204 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58595 T177 C32 P60204 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 58595 T177 C32 P60204 oass.SolrIndexSearcher.<init> Opening Searcher@6122ccff main
[junit4:junit4]   2> 58596 T177 C32 P60204 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58597 T187 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6122ccff main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4]   2> 58597 T177 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 3
[junit4:junit4]   2> 58600 T215 C36 P58508 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:43365/collection1/, StdNode: http://127.0.0.1:58508/collection1/, StdNode: http://127.0.0.1:55874/collection1/, StdNode: http://127.0.0.1:57393/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 58604 T230 C34 P55874 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58604 T197 C33 P43365 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58606 T197 C33 P43365 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 58606 T246 C35 P57393 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58607 T197 C33 P43365 oass.SolrIndexSearcher.<init> Opening Searcher@5923ec42 main
[junit4:junit4]   2> 58606 T213 C36 P58508 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 58608 T197 C33 P43365 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58607 T246 C35 P57393 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 58610 T205 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5923ec42 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4]   2> 58611 T197 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 7
[junit4:junit4]   2> 58611 T246 C35 P57393 oass.SolrIndexSearcher.<init> Opening Searcher@34499143 main
[junit4:junit4]   2> 58612 T246 C35 P57393 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58616 T256 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@34499143 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):c4)}
[junit4:junit4]   2> 58617 T246 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 12
[junit4:junit4]   2> 58623 T230 C34 P55874 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@4afd68fe lockFactory=org.apache.lucene.store.NativeFSLockFactory@5994519c),segFN=segments_3,generation=3}
[junit4:junit4]   2> 58623 T213 C36 P58508 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1b88581a lockFactory=org.apache.lucene.store.NativeFSLockFactory@4127ea39),segFN=segments_3,generation=3}
[junit4:junit4]   2> 58624 T213 C36 P58508 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 58623 T230 C34 P55874 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 58628 T213 C36 P58508 oass.SolrIndexSearcher.<init> Opening Searcher@7754fdc4 main
[junit4:junit4]   2> 58628 T230 C34 P55874 oass.SolrIndexSearcher.<init> Opening Searcher@6ce08c10 main
[junit4:junit4]   2> 58630 T213 C36 P58508 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58631 T221 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@7754fdc4 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c2 _1(5.0):c2)}
[junit4:junit4]   2> 58631 T213 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 25
[junit4:junit4]   2> 58632 T238 oasc.JmxMonitoredMap.put WARN Failed to register info bean: searcher javax.management.InstanceAlreadyExistsException: solr/collection1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
[junit4:junit4]   2> at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
[junit4:junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
[junit4:junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
[junit4:junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
[junit4:junit4]   2> at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
[junit4:junit4]   2> at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
[junit4:junit4]   2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:140)
[junit4:junit4]   2> at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:51)
[junit4:junit4]   2> at org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:305)
[junit4:junit4]   2> at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1844)
[junit4:junit4]   2> at org.apache.solr.core.SolrCore.access$000(SolrCore.java:135)
[junit4:junit4]   2> at org.apache.solr.core.SolrCore$7.call(SolrCore.java:1721)
[junit4:junit4]   2> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4]   2> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2>
[junit4:junit4]   2> 58632 T230 C34 P55874 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 58633 T238 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@6ce08c10 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):c3 _1(5.0):c1)}
[junit4:junit4]   2> 58634 T230 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 30
[junit4:junit4]   2> 58635 T215 C36 P58508 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 35
[junit4:junit4]   2> 58637 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 58640 T196 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 58644 T214 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 58647 T232 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 58650 T247 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 60652 T161 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> check const of shard1
[junit4:junit4]   2> client0
[junit4:junit4]   2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:43365_","base_url":"http://127.0.0.1:43365","leader":"true"}
[junit4:junit4]   2> 60657 T197 C33 P43365 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client1
[junit4:junit4]   2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:58508_","base_url":"http://127.0.0.1:58508"}
[junit4:junit4]   2> 60662 T215 C36 P58508 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client2
[junit4:junit4]   2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:55874_","base_url":"http://127.0.0.1:55874"}
[junit4:junit4]   2> 60666 T230 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client3
[junit4:junit4]   2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:57393_","base_url":"http://127.0.0.1:57393"}
[junit4:junit4]   2> 60670 T247 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> 60675 T231 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=1
[junit4:junit4]   2> 60686 T178 C32 P60204 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={wt=javabin&version=2} {add=[4 (1439074326991077376)]} 0 5
[junit4:junit4]   2> 60693 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:58508/collection1/]
[junit4:junit4]   2> 60694 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:58508/collection1/ against:[http://127.0.0.1:58508/collection1/] result:true
[junit4:junit4]   2> 60694 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:55874/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4]   2> 60694 T196 C33 P43365 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:57393/collection1/ against:[http://127.0.0.1:58508/collection1/] result:false
[junit4:junit4]   2> 60706 T232 C34 P55874 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4 (1439074327004708864)]} 0 7
[junit4:junit4]   2> 60706 T248 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:43365/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4 (1439074327004708864)]} 0 7
[junit4:junit4]   2> 60708 T196 C33 P43365 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={distrib.from=http://127.0.0.1:57393/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4 (1439074327004708864)]} 0 15
[junit4:junit4]   2> 60709 T249 C35 P57393 oasup.LogUpdateProcessor.finish [collection1] webapp= path=/update params={test.distrib.skip.servers=http://127.0.0.1:58508/collection1/&wt=javabin&version=2} {add=[4]} 0 19
[junit4:junit4]   2> 60710 T161 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 43365
[junit4:junit4]   2> 60734 T161 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=1430281418
[junit4:junit4]   2> 62101 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 62102 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:43365_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:43365"}
[junit4:junit4]   2> 62102 T184 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4]   2> 62106 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 62106 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 62106 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 62106 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 62106 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 62106 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 63740 T161 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 63741 T161 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 63745 T161 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@3e59c0b
[junit4:junit4]   2> 63756 T161 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 63757 T161 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 63758 T161 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 63758 T161 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2>  C33_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:43365_, base_url=http://127.0.0.1:43365, leader=true}
[junit4:junit4]   2> 63769 T161 C33 P43365 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@6f5fb626 lockFactory=org.apache.lucene.store.NativeFSLockFactory@7fd4880e),segFN=segments_3,generation=3}
[junit4:junit4]   2> 63770 T161 C33 P43365 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 63773 T161 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 63774 T161 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4]   2> 63774 T161 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index;done=false>>]
[junit4:junit4]   2> 63775 T161 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1/index
[junit4:junit4]   2> 63776 T161 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1;done=false>>]
[junit4:junit4]   2> 63776 T161 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372408195692/jetty1
[junit4:junit4]   2> 63780 T203 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 63781 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 63781 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 63781 T190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 63785 T161 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
[junit4:junit4]   2> 63785 T203 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 63786 T203 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 63785 T236 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 63787 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 63790 T183 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 63790 T219 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 63791 T190 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 63791 T253 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 63795 T219 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4]   2> 63796 T219 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4]   2> 63796 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 63796 T219 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:58508/collection1/
[junit4:junit4]   2> 63797 T219 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 START replicas=[http://127.0.0.1:43365/collection1/, http://127.0.0.1:55874/collection1/, http://127.0.0.1:57393/collection1/] nUpdates=100
[junit4:junit4]   2> 63801 T219 oasu.PeerSync.handleResponse WARN PeerSync: core=collection1 url=http://127.0.0.1:58508  couldn't connect to http://127.0.0.1:43365/collection1/, counting as success
[junit4:junit4]   2> 63802 T232 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 63802 T246 C35 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 63804 T219 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508  Received 6 versions from 127.0.0.1:55874/collection1/
[junit4:junit4]   2> 63804 T219 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:58508 Requesting updates from 127.0.0.1:55874/collection1/n=1 versions=[1439074327004708864]
[junit4:junit4]   2> 63805 T219 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508  Received 6 versions from 127.0.0.1:57393/collection1/
[junit4:junit4]   2> 63805 T219 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:58508  Our versions are newer. ourLowThreshold=1439074324562575360 otherHigh=1439074324661141504
[junit4:junit4]   2> 63808 T230 C34 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={distrib=false&getUpdates=1439074327004708864&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 63815 T219 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1439074327004708864)]} 0 6
[junit4:junit4]   2> 63815 T219 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:58508 DONE. sync succeeded
[junit4:junit4]   2> 63815 T219 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 63816 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: try and ask http://127.0.0.1:43365/collection1/ to sync
[junit4:junit4]   2> 63816 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: try and ask http://127.0.0.1:55874/collection1/ to sync
[junit4:junit4]   2> 63817 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: try and ask http://127.0.0.1:57393/collection1/ to sync
[junit4:junit4]   2> 63818 T219 oasc.SolrException.log ERROR Sync request error: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:43365/collection1
[junit4:junit4]   2> 63819 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/: Sync failed - asking replica (http://127.0.0.1:43365/collection1/) to recover.
[junit4:junit4]   2> 63821 T231 C34 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 START replicas=[http://127.0.0.1:58508/collection1/] nUpdates=100
[junit4:junit4]   2> 63821 T248 C35 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 START replicas=[http://127.0.0.1:58508/collection1/] nUpdates=100
[junit4:junit4]   2> 63845 T293 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 63846 T161 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 43365
[junit4:junit4]   2> ASYNC  NEW_CORE C37 name=collection1 org.apache.solr.core.SolrCore@3266f7c5 url=http://127.0.0.1:58508/collection1 node=127.0.0.1:58508_ C37_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:58508_, base_url=http://127.0.0.1:58508}
[junit4:junit4]   2> 63848 T213 C37 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 63848 T215 C37 P58508 oasc.SolrCore.execute [collection1] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> ASYNC  NEW_CORE C38 name=collection1 org.apache.solr.core.SolrCore@151671f7 url=http://127.0.0.1:55874/collection1 node=127.0.0.1:55874_ C38_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:55874_, base_url=http://127.0.0.1:55874}
[junit4:junit4]   2> 63849 T231 C38 P55874 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:55874  Received 6 versions from 127.0.0.1:58508/collection1/
[junit4:junit4]   2> ASYNC  NEW_CORE C39 name=collection1 org.apache.solr.core.SolrCore@6c18b5da url=http://127.0.0.1:57393/collection1 node=127.0.0.1:57393_ C39_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:57393_, base_url=http://127.0.0.1:57393}
[junit4:junit4]   2> 63850 T248 C39 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393  Received 6 versions from 127.0.0.1:58508/collection1/
[junit4:junit4]   2> 63850 T231 C38 P55874 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:55874  Our versions are newer. ourLowThreshold=1439074324584595456 otherHigh=1439074324661141504
[junit4:junit4]   2> 63851 T248 C39 P57393 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:57393  Our versions are newer. ourLowThreshold=1439074324584595456 otherHigh=1439074324661141504
[junit4:junit4]   2> 63852 T248 C39 P57393 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:57393 DONE. sync succeeded
[junit4:junit4]   2> 63851 T231 C38 P55874 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:55874 DONE. sync succeeded
[junit4:junit4]   2> 63853 T248 C39 P57393 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:58508/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=32
[junit4:junit4]   2> 63853 T231 C38 P55874 oasc.SolrCore.execute [collection1] webapp= path=/get params={sync=http://127.0.0.1:58508/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=33
[junit4:junit4]   2> 63854 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/:  sync completed with http://127.0.0.1:57393/collection1/
[junit4:junit4]   2> 63855 T219 oasc.SyncStrategy.syncToMe http://127.0.0.1:58508/collection1/:  sync completed with http://127.0.0.1:55874/collection1/
[junit4:junit4]   2> 63856 T219 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:58508/collection1/ shard1
[junit4:junit4]   2> 63856 T219 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 63857 T293 oasc.SolrException.log ERROR http://127.0.0.1:58508/collection1/: Could not tell a replica to recover:org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:43365
[junit4:junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
[junit4:junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
[junit4:junit4]   2> at org.apache.solr.cloud.SyncStrategy$1.run(SyncStrategy.java:297)
[junit4:junit4]   2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit4:junit4]   2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2> at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:43365 refused
[junit4:junit4]   2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
[junit4:junit4]   2> at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
[junit4:junit4]   2> at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
[junit4:junit4]   2> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
[junit4:junit4]   2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
[junit4:junit4]   2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
[junit4:junit4]   2> at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
[junit4:junit4]   2> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
[junit4:junit4]   2> ... 5 more
[junit4:junit4]   2> Caused by: java.net.ConnectException: Connection refused
[junit4:junit4]   2> at java.net.PlainSocketImpl.socketConnect(Native Method)
[junit4:junit4]   2> at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
[junit4:junit4]   2> at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
[junit4:junit4]   2> at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
[junit4:junit4]   2> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
[junit4:junit4]   2> at java.net.Socket.connect(Socket.java:579)
[junit4:junit4]   2> at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
[junit4:junit4]   2> at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
[junit4:junit4]   2> ... 12 more
[junit4:junit4]   2>
[junit4:junit4]   2> 63867 T219 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 63871 T219 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 65116 T184 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 65117 T184 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:43365_",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:43365"}
[junit4:junit4]   2> 65130 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 65130 T183 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 65130 T253 oascc.ZkStateReader$2.process A cluster

[...truncated too long message...]

:"127.0.0.1:43365_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:43365"},
[junit4:junit4]   1>                 "core_node2":{
[junit4:junit4]   1>                   "state":"down",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:58508_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:58508"},
[junit4:junit4]   1>                 "core_node3":{
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:55874_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:55874"},
[junit4:junit4]   1>                 "core_node4":{
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:57393_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:57393",
[junit4:junit4]   1>                   "leader":"true"}}}},
[junit4:junit4]   1>           "router":"compositeId"},
[junit4:junit4]   1>         "control_collection":{
[junit4:junit4]   1>           "shards":{"shard1":{
[junit4:junit4]   1>               "range":"80000000-7fffffff",
[junit4:junit4]   1>               "state":"active",
[junit4:junit4]   1>               "replicas":{"core_node1":{
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:60204_",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:60204",
[junit4:junit4]   1>                   "leader":"true"}}}},
[junit4:junit4]   1>           "router":"compositeId"}}
[junit4:junit4]   1>   /solr/aliases.json (0)
[junit4:junit4]   1>   /solr/live_nodes (4)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:55874_ (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:57393_ (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:43365_ (0)
[junit4:junit4]   1>    /solr/live_nodes/127.0.0.1:60204_ (0)
[junit4:junit4]   1>   /solr/overseer (3)
[junit4:junit4]   1>   DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1>    /solr/overseer/queue (0)
[junit4:junit4]   1>    /solr/overseer/queue-work (0)
[junit4:junit4]   1>    /solr/overseer/collection-queue-work (0)
[junit4:junit4]   1>   /solr/collections (2)
[junit4:junit4]   1>    /solr/collections/collection1 (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/collection1/shards (0)
[junit4:junit4]   1>     /solr/collections/collection1/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/collection1/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/collection1/leader_elect/shard1/election (3)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89942143517917196-core_node4-n_0000000003 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89942143517917198-core_node1-n_0000000004 (0)
[junit4:junit4]   1>        /solr/collections/collection1/leader_elect/shard1/election/89942143517917194-core_node3-n_0000000005 (0)
[junit4:junit4]   1>     /solr/collections/collection1/leaders (1)
[junit4:junit4]   1>      /solr/collections/collection1/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "core":"collection1",
[junit4:junit4]   1>            "node_name":"127.0.0.1:57393_",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:57393"}
[junit4:junit4]   1>    /solr/collections/control_collection (3)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"configName":"conf1"}
[junit4:junit4]   1>     /solr/collections/control_collection/shards (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leader_elect (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leader_elect/shard1 (1)
[junit4:junit4]   1>       /solr/collections/control_collection/leader_elect/shard1/election (1)
[junit4:junit4]   1>        /solr/collections/control_collection/leader_elect/shard1/election/89942143517917187-core_node1-n_0000000000 (0)
[junit4:junit4]   1>     /solr/collections/control_collection/leaders (1)
[junit4:junit4]   1>      /solr/collections/control_collection/leaders/shard1 (0)
[junit4:junit4]   1>      DATA:
[junit4:junit4]   1>          {
[junit4:junit4]   1>            "core":"collection1",
[junit4:junit4]   1>            "node_name":"127.0.0.1:60204_",
[junit4:junit4]   1>            "base_url":"http://127.0.0.1:60204"}
[junit4:junit4]   1>   /solr/overseer_elect (2)
[junit4:junit4]   1>    /solr/overseer_elect/election (4)
[junit4:junit4]   1>     /solr/overseer_elect/election/89942143517917187-127.0.0.1:60204_-n_0000000000 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89942143517917198-127.0.0.1:43365_-n_0000000005 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89942143517917196-127.0.0.1:57393_-n_0000000004 (0)
[junit4:junit4]   1>     /solr/overseer_elect/election/89942143517917194-127.0.0.1:55874_-n_0000000003 (0)
[junit4:junit4]   1>    /solr/overseer_elect/leader (0)
[junit4:junit4]   1>    DATA:
[junit4:junit4]   1>        {"id":"89942143517917187-127.0.0.1:60204_-n_0000000000"}
[junit4:junit4]   1>  /zookeeper (1)
[junit4:junit4]   1>  DATA:
[junit4:junit4]   1>      
[junit4:junit4]   1>
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=SyncSliceTest -Dtests.method=testDistribSearch -Dtests.seed=EE579D43B894FBE1 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=iw -Dtests.timezone=Asia/Pontianak -Dtests.file.encoding=US-ASCII
[junit4:junit4] FAILURE  106s J0 | SyncSliceTest.testDistribSearch <<<
[junit4:junit4]    > Throwable #1: java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:43365/collection1lastClient and got 256 from http://127.0.0.1:55874/collection1
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([EE579D43B894FBE1:6FB1135BCFCB9BDD]:0)
[junit4:junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
[junit4:junit4]    > at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
[junit4:junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
[junit4:junit4]    > at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 137676 T161 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 106362 T160 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 3 leaked thread(s).
[junit4:junit4]   2> 137799 T236 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 137800 T236 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 137800 T236 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=Direct), text=PostingsFormat(name=Memory doPackFST= true), _version_=PostingsFormat(name=Direct), rnd_b=PostingsFormat(name=Memory doPackFST= false), intDefault=PostingsFormat(name=Direct), timestamp=PostingsFormat(name=Direct), id=PostingsFormat(name=Memory doPackFST= false), a_t=PostingsFormat(name=Direct), range_facet_sl=PostingsFormat(name=Memory doPackFST= true), range_facet_si=PostingsFormat(name=SimpleText), other_tl1=PostingsFormat(name=Direct), multiDefault=PostingsFormat(name=SimpleText), a_si=PostingsFormat(name=SimpleText)}, docValues:{timestamp=DocValuesFormat(name=SimpleText)}, sim=RandomSimilarityProvider(queryNorm=true,coord=no): {}, locale=iw, timezone=Asia/Pontianak
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=1,free=53343576,total=126681088
[junit4:junit4]   2> NOTE: All tests run in this JVM: [ScriptEngineTest, DirectUpdateHandlerOptimizeTest, ParsingFieldUpdateProcessorsTest, TestCharFilters, TestSchemaVersionResource, RAMDirectoryFactoryTest, TestLRUCache, TestAddFieldRealTimeGet, TestDynamicFieldCollectionResource, TestSolrQueryParser, PeerSyncTest, CacheHeaderTest, UpdateRequestProcessorFactoryTest, SoftAutoCommitTest, PrimitiveFieldTypeTest, TestFastLRUCache, AssignTest, SyncSliceTest]
[junit4:junit4] Completed on J0 in 107.40s, 1 test, 1 failure <<< FAILURES!

[...truncated 874 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 1 failure, 20 ignored (13 assumptions)

Total time: 37 minutes 56 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.7.0_25) - Build # 6348 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6348/
Java: 32bit/jdk1.7.0_25 -client -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:     1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:724)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
   1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
        at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeySafeLeaderTest

Error Message:
There are still zombie threads that couldn't be terminated:    1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]         at sun.misc.Unsafe.park(Native Method)         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:724)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
   1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)
        at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)




Build Log:
[...truncated 9507 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.ChaosMonkeySafeLeaderTest
[junit4:junit4]   2> 612255 T2992 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /x_/rw
[junit4:junit4]   2> 612266 T2992 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./solrtest-ChaosMonkeySafeLeaderTest-1372413435935
[junit4:junit4]   2> 612267 T2992 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 612267 T2993 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 612367 T2992 oasc.ZkTestServer.run start zk server on port:51741
[junit4:junit4]   2> 612368 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 612577 T2999 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7c24fe name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 612578 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 612578 T2992 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 612583 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 612584 T3001 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f0fc30 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 612584 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 612584 T2992 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 612586 T2992 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 612588 T2992 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 612589 T2992 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 612591 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 612592 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 612595 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 612595 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 612598 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 612598 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 612600 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 612600 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 612602 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 612603 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 612605 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 612605 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 612607 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 612607 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 612609 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 612609 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 612611 T2992 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 612611 T2992 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 612709 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 612723 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45695
[junit4:junit4]   2> 612724 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 612724 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 612725 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281
[junit4:junit4]   2> 612725 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/solr.xml
[junit4:junit4]   2> 612725 T2992 oasc.CoreContainer.<init> New CoreContainer 16159686
[junit4:junit4]   2> 612726 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/'
[junit4:junit4]   2> 612726 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/'
[junit4:junit4]   2> 612792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 612792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 612793 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 612793 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 612793 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 612794 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 612794 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 612794 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 612795 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 612795 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 612798 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 612799 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 612799 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 612800 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 612801 T3012 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14f3ce8 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 612801 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 612803 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 612806 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 612807 T3014 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@eac7a7 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 612807 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 612809 T2992 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 612811 T2992 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 612813 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 612815 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45695_x_%2Frw
[junit4:junit4]   2> 612815 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45695_x_%2Frw
[junit4:junit4]   2> 612818 T2992 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 612821 T2992 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 612823 T2992 oasc.Overseer.start Overseer (id=89942486937567235-127.0.0.1:45695_x_%2Frw-n_0000000000) starting
[junit4:junit4]   2> 612826 T2992 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 612828 T3016 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 612829 T2992 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 612830 T2992 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 612832 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 612834 T3015 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 612836 T3017 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 612837 T3017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 612838 T3017 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 614337 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 614338 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:45695_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:45695/x_/rw"}
[junit4:junit4]   2> 614338 T3015 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 614338 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 614341 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 614838 T3017 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 614838 T3017 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1
[junit4:junit4]   2> 614839 T3017 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 614839 T3017 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 614840 T3017 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 614841 T3017 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/'
[junit4:junit4]   2> 614841 T3017 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/lib/README' to classloader
[junit4:junit4]   2> 614842 T3017 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 614870 T3017 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 614907 T3017 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 614909 T3017 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 614913 T3017 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 615303 T3017 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 615309 T3017 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 615312 T3017 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 615321 T3017 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 615324 T3017 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 615327 T3017 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 615327 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 615328 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 615328 T3017 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 615329 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 615329 T3017 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 615329 T3017 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 615329 T3017 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-controljetty-1372413436281/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/
[junit4:junit4]   2> 615329 T3017 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 615330 T3017 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data
[junit4:junit4]   2> 615330 T3017 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index/
[junit4:junit4]   2> 615331 T3017 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 615331 T3017 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index
[junit4:junit4]   2> 615334 T3017 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/control/data/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 615334 T3017 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 615335 T3017 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 615336 T3017 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 615336 T3017 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 615337 T3017 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 615338 T3017 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 615338 T3017 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 615340 T3017 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 615343 T3017 oass.SolrIndexSearcher.<init> Opening Searcher@1c81327 main
[junit4:junit4]   2> 615344 T3017 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 615344 T3017 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 615347 T3018 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1c81327 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 615349 T3017 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 615349 T3017 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45695/x_/rw collection:control_collection shard:shard1
[junit4:junit4]   2> 615350 T3017 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 615354 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 615356 T3017 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 615356 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 615356 T3017 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45695/x_/rw/collection1/
[junit4:junit4]   2> 615356 T3017 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 615357 T3017 oasc.SyncStrategy.syncToMe http://127.0.0.1:45695/x_/rw/collection1/ has no replicas
[junit4:junit4]   2> 615357 T3017 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45695/x_/rw/collection1/ shard1
[junit4:junit4]   2> 615357 T3017 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 615844 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 615850 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 615862 T3017 oasc.ZkController.register We are http://127.0.0.1:45695/x_/rw/collection1/ and leader is http://127.0.0.1:45695/x_/rw/collection1/
[junit4:junit4]   2> 615863 T3017 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45695/x_/rw
[junit4:junit4]   2> 615863 T3017 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 615863 T3017 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 615863 T3017 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 615866 T3017 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 615867 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 615867 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 615868 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 615871 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 615873 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 615874 T3021 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1436fa2 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 615874 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 615876 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 615877 T2992 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 615935 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 615937 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33049
[junit4:junit4]   2> 615938 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 615938 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 615938 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545
[junit4:junit4]   2> 615939 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/solr.xml
[junit4:junit4]   2> 615939 T2992 oasc.CoreContainer.<init> New CoreContainer 22280971
[junit4:junit4]   2> 615939 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/'
[junit4:junit4]   2> 615940 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/'
[junit4:junit4]   2> 616002 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 616002 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 616002 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 616003 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 616004 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 616004 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 616004 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 616007 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 616008 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 616008 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 616009 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 616010 T3032 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1e1661c name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 616011 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 616013 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 616015 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 616016 T3034 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@14b1806 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 616017 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 616019 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 617022 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33049_x_%2Frw
[junit4:junit4]   2> 617023 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33049_x_%2Frw
[junit4:junit4]   2> 617026 T3014 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 617026 T3021 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 617026 T3034 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 617027 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 617031 T3035 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 617032 T3035 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 617033 T3035 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 617354 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 617354 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:45695_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:45695/x_/rw"}
[junit4:junit4]   2> 617357 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:33049_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:33049/x_/rw"}
[junit4:junit4]   2> 617358 T3015 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 3
[junit4:junit4]   2> 617358 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 617361 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 617361 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 617361 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 618033 T3035 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 618034 T3035 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1
[junit4:junit4]   2> 618034 T3035 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 618035 T3035 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 618036 T3035 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 618038 T3035 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/'
[junit4:junit4]   2> 618039 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/lib/README' to classloader
[junit4:junit4]   2> 618039 T3035 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 618085 T3035 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 618122 T3035 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 618123 T3035 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 618127 T3035 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 618550 T3035 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 618556 T3035 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 618559 T3035 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 618569 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 618572 T3035 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 618575 T3035 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 618576 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 618576 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 618576 T3035 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 618577 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 618577 T3035 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 618578 T3035 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 618578 T3035 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty1-1372413439545/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/
[junit4:junit4]   2> 618578 T3035 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 618579 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1
[junit4:junit4]   2> 618579 T3035 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index/
[junit4:junit4]   2> 618579 T3035 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 618580 T3035 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index
[junit4:junit4]   2> 618582 T3035 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 618583 T3035 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 618584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 618584 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 618585 T3035 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 618586 T3035 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 618586 T3035 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 618586 T3035 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 618587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 618587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 618587 T3035 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 618590 T3035 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 618593 T3035 oass.SolrIndexSearcher.<init> Opening Searcher@18a64f3 main
[junit4:junit4]   2> 618593 T3035 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 618593 T3035 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 618602 T3036 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18a64f3 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 618605 T3035 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 618605 T3035 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33049/x_/rw collection:collection1 shard:shard1
[junit4:junit4]   2> 618606 T3035 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 618612 T3035 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 618614 T3035 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 618614 T3035 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 618614 T3035 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4]   2> 618615 T3035 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 618615 T3035 oasc.SyncStrategy.syncToMe http://127.0.0.1:33049/x_/rw/collection1/ has no replicas
[junit4:junit4]   2> 618615 T3035 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:33049/x_/rw/collection1/ shard1
[junit4:junit4]   2> 618616 T3035 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 618866 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 618872 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 618872 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 618872 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 618921 T3035 oasc.ZkController.register We are http://127.0.0.1:33049/x_/rw/collection1/ and leader is http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4]   2> 618921 T3035 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33049/x_/rw
[junit4:junit4]   2> 618921 T3035 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 618922 T3035 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 618922 T3035 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 618924 T3035 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 618924 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 618925 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 618925 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 618990 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 618992 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40786
[junit4:junit4]   2> 618993 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 618993 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 618993 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596
[junit4:junit4]   2> 618994 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/solr.xml
[junit4:junit4]   2> 618994 T2992 oasc.CoreContainer.<init> New CoreContainer 5234384
[junit4:junit4]   2> 618994 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/'
[junit4:junit4]   2> 618995 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/'
[junit4:junit4]   2> 619045 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 619045 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 619046 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 619046 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 619046 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 619047 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 619048 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 619063 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 619064 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 619064 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 619065 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 619066 T3048 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ac279 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 619067 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 619068 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 619071 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 619072 T3050 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da84e1 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 619072 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 619075 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 620147 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40786_x_%2Frw
[junit4:junit4]   2> 620449 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40786_x_%2Frw
[junit4:junit4]   2> 620460 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 620464 T3014 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 620466 T3021 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 620469 T3034 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 620467 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 620471 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 620476 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 620480 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:33049_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:33049/x_/rw"}
[junit4:junit4]   2> 620494 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 620494 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 620503 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 620710 T3050 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 620711 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 620930 T3051 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 620934 T3051 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 620937 T3051 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 622014 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 622026 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:40786_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:40786/x_/rw"}
[junit4:junit4]   2> 622036 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 622036 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 622049 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 622049 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 622051 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 622051 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 622939 T3051 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 622964 T3051 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1
[junit4:junit4]   2> 622964 T3051 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 622977 T3051 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 622977 T3051 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 622986 T3051 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/'
[junit4:junit4]   2> 622999 T3051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/lib/README' to classloader
[junit4:junit4]   2> 623000 T3051 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 623714 T3051 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 624005 T3051 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 624012 T3051 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 624026 T3051 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 624892 T3051 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 624899 T3051 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 624902 T3051 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 624917 T3051 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 624921 T3051 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 624924 T3051 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 624926 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 624928 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 624928 T3051 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 624929 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 624929 T3051 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 624930 T3051 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 624931 T3051 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty2-1372413442596/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/
[junit4:junit4]   2> 624932 T3051 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 624935 T3051 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2
[junit4:junit4]   2> 624936 T3051 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index/
[junit4:junit4]   2> 624936 T3051 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 624936 T3051 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index
[junit4:junit4]   2> 624958 T3051 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 624958 T3051 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 624962 T3051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 624962 T3051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 624963 T3051 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 624965 T3051 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 624966 T3051 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 624966 T3051 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 624967 T3051 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 624968 T3051 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 624968 T3051 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 624972 T3051 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 624978 T3051 oass.SolrIndexSearcher.<init> Opening Searcher@1ad25f9 main
[junit4:junit4]   2> 624980 T3051 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 624980 T3051 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 625053 T3053 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1ad25f9 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 625057 T3051 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 625057 T3051 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40786/x_/rw collection:collection1 shard:shard2
[junit4:junit4]   2> 625058 T3051 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
[junit4:junit4]   2> 625064 T3051 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
[junit4:junit4]   2> 625067 T3051 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 625067 T3051 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 625067 T3051 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40786/x_/rw/collection1/
[junit4:junit4]   2> 625067 T3051 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 625067 T3051 oasc.SyncStrategy.syncToMe http://127.0.0.1:40786/x_/rw/collection1/ has no replicas
[junit4:junit4]   2> 625068 T3051 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40786/x_/rw/collection1/ shard2
[junit4:junit4]   2> 625068 T3051 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
[junit4:junit4]   2> 625070 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 625076 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 625076 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 625076 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 625076 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 625123 T3051 oasc.ZkController.register We are http://127.0.0.1:40786/x_/rw/collection1/ and leader is http://127.0.0.1:40786/x_/rw/collection1/
[junit4:junit4]   2> 625123 T3051 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40786/x_/rw
[junit4:junit4]   2> 625123 T3051 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 625124 T3051 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 625124 T3051 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 625126 T3051 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 625129 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 625130 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 625131 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 625243 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 625260 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40386
[junit4:junit4]   2> 625261 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 625261 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 625262 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811
[junit4:junit4]   2> 625262 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/solr.xml
[junit4:junit4]   2> 625262 T2992 oasc.CoreContainer.<init> New CoreContainer 686762
[junit4:junit4]   2> 625263 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/'
[junit4:junit4]   2> 625263 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/'
[junit4:junit4]   2> 625319 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 625320 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 625320 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 625320 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 625321 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 625321 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 625321 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 625322 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 625322 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 625323 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 625340 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 625341 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 625342 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 625344 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 625349 T3065 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@121f2c7 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 625349 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 625350 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 625356 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 625357 T3067 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@84839e name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 625357 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 625360 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 626364 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40386_x_%2Frw
[junit4:junit4]   2> 626365 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40386_x_%2Frw
[junit4:junit4]   2> 626366 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 626374 T3021 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 626374 T3067 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 626374 T3014 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 626374 T3050 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 626374 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626375 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626374 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626375 T3034 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 626379 T3068 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 626379 T3068 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 626380 T3068 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 626584 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 626585 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:40786_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:40786/x_/rw"}
[junit4:junit4]   2> 626588 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:40386_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:40386/x_/rw"}
[junit4:junit4]   2> 626588 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 626588 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4:junit4]   2> 626591 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626591 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626591 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626591 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 626591 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 627380 T3068 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 627381 T3068 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1
[junit4:junit4]   2> 627381 T3068 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 627382 T3068 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 627382 T3068 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 627384 T3068 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/'
[junit4:junit4]   2> 627386 T3068 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/lib/README' to classloader
[junit4:junit4]   2> 627386 T3068 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 627438 T3068 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 627477 T3068 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 627478 T3068 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 627483 T3068 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 627897 T3068 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 627903 T3068 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 627906 T3068 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 627918 T3068 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 627923 T3068 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 627928 T3068 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 627930 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 627930 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 627930 T3068 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 627931 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 627931 T3068 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 627932 T3068 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 627932 T3068 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty3-1372413448811/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/
[junit4:junit4]   2> 627932 T3068 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 627933 T3068 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3
[junit4:junit4]   2> 627934 T3068 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index/
[junit4:junit4]   2> 627934 T3068 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 627935 T3068 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index
[junit4:junit4]   2> 627938 T3068 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 627939 T3068 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 627941 T3068 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 627941 T3068 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 627941 T3068 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 627942 T3068 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 627943 T3068 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 627943 T3068 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 627943 T3068 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 627944 T3068 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 627944 T3068 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 627947 T3068 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 627951 T3068 oass.SolrIndexSearcher.<init> Opening Searcher@1e6ce72 main
[junit4:junit4]   2> 627952 T3068 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 627952 T3068 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 627957 T3069 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@1e6ce72 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 627960 T3068 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 627960 T3068 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:40386/x_/rw collection:collection1 shard:shard3
[junit4:junit4]   2> 627960 T3068 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard3/election
[junit4:junit4]   2> 627964 T3068 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard3
[junit4:junit4]   2> 627966 T3068 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 627966 T3068 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 627966 T3068 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:40386/x_/rw/collection1/
[junit4:junit4]   2> 627967 T3068 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 627967 T3068 oasc.SyncStrategy.syncToMe http://127.0.0.1:40386/x_/rw/collection1/ has no replicas
[junit4:junit4]   2> 627967 T3068 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:40386/x_/rw/collection1/ shard3
[junit4:junit4]   2> 627968 T3068 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard3
[junit4:junit4]   2> 628098 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 628102 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 628102 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 628102 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 628102 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 628102 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 628121 T3068 oasc.ZkController.register We are http://127.0.0.1:40386/x_/rw/collection1/ and leader is http://127.0.0.1:40386/x_/rw/collection1/
[junit4:junit4]   2> 628121 T3068 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:40386/x_/rw
[junit4:junit4]   2> 628121 T3068 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 628122 T3068 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 628122 T3068 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 628124 T3068 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 628125 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 628125 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 628125 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 628203 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 628212 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42828
[junit4:junit4]   2> 628212 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 628213 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 628213 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797
[junit4:junit4]   2> 628213 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/solr.xml
[junit4:junit4]   2> 628214 T2992 oasc.CoreContainer.<init> New CoreContainer 17887150
[junit4:junit4]   2> 628214 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/'
[junit4:junit4]   2> 628214 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/'
[junit4:junit4]   2> 628268 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 628269 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 628269 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 628270 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 628270 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 628270 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 628271 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 628271 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 628271 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 628272 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 628275 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 628275 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 628275 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 628276 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 628278 T3081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e0d861 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 628279 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 628280 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 628297 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 628298 T3083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f1254c name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 628298 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 628301 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 629303 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42828_x_%2Frw
[junit4:junit4]   2> 629305 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42828_x_%2Frw
[junit4:junit4]   2> 629306 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 629306 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 629312 T3021 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 629312 T3014 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 629312 T3050 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 629312 T3083 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 629312 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629312 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629312 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629313 T3067 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 629313 T3034 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 629318 T3084 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 629319 T3084 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 629319 T3084 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 629608 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 629609 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard3",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:40386_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:40386/x_/rw"}
[junit4:junit4]   2> 629613 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:42828_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:42828/x_/rw"}
[junit4:junit4]   2> 629613 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 629613 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 629616 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629616 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629616 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629616 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629616 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 629616 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 630320 T3084 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 630320 T3084 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1
[junit4:junit4]   2> 630320 T3084 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 630321 T3084 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 630322 T3084 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 630323 T3084 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/'
[junit4:junit4]   2> 630323 T3084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/lib/README' to classloader
[junit4:junit4]   2> 630324 T3084 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 630358 T3084 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 630395 T3084 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 630396 T3084 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 630400 T3084 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 630806 T3084 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 630812 T3084 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 630815 T3084 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 630824 T3084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 630827 T3084 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 630830 T3084 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 630831 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 630831 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 630831 T3084 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 630832 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 630832 T3084 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 630832 T3084 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 630833 T3084 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty4-1372413451797/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/
[junit4:junit4]   2> 630833 T3084 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 630833 T3084 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4
[junit4:junit4]   2> 630834 T3084 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index/
[junit4:junit4]   2> 630834 T3084 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 630834 T3084 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index
[junit4:junit4]   2> 630837 T3084 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty4/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 630837 T3084 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 630839 T3084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 630839 T3084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 630839 T3084 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 630840 T3084 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 630840 T3084 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 630841 T3084 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 630841 T3084 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 630841 T3084 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 630842 T3084 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 630844 T3084 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 630847 T3084 oass.SolrIndexSearcher.<init> Opening Searcher@c4a32c main
[junit4:junit4]   2> 630848 T3084 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 630848 T3084 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 630853 T3085 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@c4a32c main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 630855 T3084 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 630855 T3084 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42828/x_/rw collection:collection1 shard:shard1
[junit4:junit4]   2> 630858 T3084 oasc.ZkController.register We are http://127.0.0.1:42828/x_/rw/collection1/ and leader is http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4]   2> 630858 T3084 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42828/x_/rw
[junit4:junit4]   2> 630859 T3084 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 630859 T3084 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C493 name=collection1 org.apache.solr.core.SolrCore@1feff0f url=http://127.0.0.1:42828/x_/rw/collection1 node=127.0.0.1:42828_x_%2Frw C493_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:42828_x_%2Frw, base_url=http://127.0.0.1:42828/x_/rw}
[junit4:junit4]   2> 630859 T3086 C493 P42828 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 630860 T3084 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 630864 T3086 C493 P42828 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 630865 T3086 C493 P42828 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 630865 T3086 C493 P42828 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 630866 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 630866 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 630866 T3086 C493 P42828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 630867 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 630877 T3028 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 630932 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 630936 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42858
[junit4:junit4]   2> 630936 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 630937 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 630937 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537
[junit4:junit4]   2> 630937 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/solr.xml
[junit4:junit4]   2> 630938 T2992 oasc.CoreContainer.<init> New CoreContainer 3800004
[junit4:junit4]   2> 630938 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/'
[junit4:junit4]   2> 630938 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/'
[junit4:junit4]   2> 630989 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 630990 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 630990 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 630990 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 630991 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 630992 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 630992 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 630995 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 630996 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 630996 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 630997 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 630998 T3098 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@cb10b0 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 630999 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 631000 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 631004 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 631005 T3100 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1719f84 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 631005 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 631008 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 631122 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 631123 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:42828_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:42828/x_/rw"}
[junit4:junit4]   2> 631126 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631126 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631126 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631126 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631126 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631126 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631126 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 631879 T3028 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 631880 T3028 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:42828_x_%252Frw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1003
[junit4:junit4]   2> 632011 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42858_x_%2Frw
[junit4:junit4]   2> 632012 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42858_x_%2Frw
[junit4:junit4]   2> 632014 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 632014 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 632015 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 632015 T3050 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632015 T3021 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632016 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632015 T3014 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632016 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632016 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632017 T3100 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632016 T3067 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632017 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632018 T3083 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632018 T3034 oascc.ZkStateReader$3.process Updating live nodes... (6)
[junit4:junit4]   2> 632042 T3101 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 632042 T3101 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 632043 T3101 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 632631 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 632632 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node5",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:42858_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:42858/x_/rw"}
[junit4:junit4]   2> 632632 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 632632 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
[junit4:junit4]   2> 632637 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632637 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632637 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632637 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632637 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632637 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 632637 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 633043 T3101 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 633044 T3101 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1
[junit4:junit4]   2> 633045 T3101 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 633046 T3101 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 633046 T3101 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 633048 T3101 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/'
[junit4:junit4]   2> 633049 T3101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/lib/README' to classloader
[junit4:junit4]   2> 633050 T3101 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 633110 T3101 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 633152 T3101 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 633153 T3101 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 633157 T3101 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 633590 T3101 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 633596 T3101 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 633598 T3101 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 633608 T3101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 633611 T3101 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 633614 T3101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 633615 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 633615 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 633615 T3101 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 633616 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 633616 T3101 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 633616 T3101 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 633617 T3101 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty5-1372413454537/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/
[junit4:junit4]   2> 633617 T3101 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 633617 T3101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5
[junit4:junit4]   2> 633618 T3101 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index/
[junit4:junit4]   2> 633618 T3101 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 633618 T3101 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index
[junit4:junit4]   2> 633768 T3101 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty5/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 633768 T3101 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 633770 T3101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 633770 T3101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 633771 T3101 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 633771 T3101 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 633772 T3101 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 633772 T3101 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 633773 T3101 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 633773 T3101 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 633774 T3101 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 633776 T3101 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 633779 T3101 oass.SolrIndexSearcher.<init> Opening Searcher@19032b8 main
[junit4:junit4]   2> 633779 T3101 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 633780 T3101 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 633785 T3102 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@19032b8 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 633788 T3101 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 633788 T3101 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42858/x_/rw collection:collection1 shard:shard2
[junit4:junit4]   2> 633791 T3101 oasc.ZkController.register We are http://127.0.0.1:42858/x_/rw/collection1/ and leader is http://127.0.0.1:40786/x_/rw/collection1/
[junit4:junit4]   2> 633791 T3101 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42858/x_/rw
[junit4:junit4]   2> 633791 T3101 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 633791 T3101 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> 633792 T3101 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C494 name=collection1 org.apache.solr.core.SolrCore@1ffb77f url=http://127.0.0.1:42858/x_/rw/collection1 node=127.0.0.1:42858_x_%2Frw C494_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:42858_x_%2Frw, base_url=http://127.0.0.1:42858/x_/rw}
[junit4:junit4]   2> 633793 T3103 C494 P42858 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 633799 T3103 C494 P42858 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 633799 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 633799 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 633799 T3103 C494 P42858 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 633800 T3103 C494 P42858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 633800 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 633801 T3103 C494 P42858 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 633805 T3044 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node5, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 633864 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 633877 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33030
[junit4:junit4]   2> 633878 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 633878 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 633878 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471
[junit4:junit4]   2> 633879 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/solr.xml
[junit4:junit4]   2> 633879 T2992 oasc.CoreContainer.<init> New CoreContainer 25622568
[junit4:junit4]   2> 633879 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/'
[junit4:junit4]   2> 633880 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/'
[junit4:junit4]   2> ASYNC  NEW_CORE C495 name=collection1 org.apache.solr.core.SolrCore@1feff0f url=http://127.0.0.1:42828/x_/rw/collection1 node=127.0.0.1:42828_x_%2Frw C495_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:42828_x_%2Frw, base_url=http://127.0.0.1:42828/x_/rw}
[junit4:junit4]   2> 633882 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:33049/x_/rw/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 633883 T3086 C495 P42828 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42828/x_/rw START replicas=[http://127.0.0.1:33049/x_/rw/collection1/] nUpdates=100
[junit4:junit4]   2> 633883 T3086 C495 P42828 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 633883 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 633884 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 633884 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 633884 T3086 C495 P42828 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 633884 T3086 C495 P42828 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:33049/x_/rw/collection1/. core=collection1
[junit4:junit4]   2> 633884 T3086 C495 P42828 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C496 name=collection1 org.apache.solr.core.SolrCore@2ebc10 url=http://127.0.0.1:33049/x_/rw/collection1 node=127.0.0.1:33049_x_%2Frw C496_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:33049_x_%2Frw, base_url=http://127.0.0.1:33049/x_/rw, leader=true}
[junit4:junit4]   2> 633896 T3030 C496 P33049 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4]   2> 633899 T3028 C496 P33049 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 633902 T3028 C496 P33049 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty1/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 633909 T3028 C496 P33049 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 633909 T3028 C496 P33049 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 633910 T3028 C496 P33049 oass.SolrIndexSearcher.<init> Opening Searcher@1a024f9 realtime
[junit4:junit4]   2> 633911 T3028 C496 P33049 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 633911 T3028 C496 P33049 oasup.LogUpdateProcessor.finish [collection1] webapp=/x_/rw path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 14
[junit4:junit4]   2> 633912 T3086 C495 P42828 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 633912 T3086 C495 P42828 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 633916 T3027 C496 P33049 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 633916 T3027 C496 P33049 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4]   2> 633917 T3086 C495 P42828 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 633917 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 633918 T3086 C495 P42828 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 633918 T3086 C495 P42828 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 633919 T3086 C495 P42828 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 633970 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 633971 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 633972 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 633972 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 633973 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 633973 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 633974 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 633977 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 633977 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 633978 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 633978 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 633980 T3116 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@143a012 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 633981 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 633986 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 633991 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 633992 T3118 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6152e5 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 633993 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 633996 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 634142 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 634143 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node5",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:42858_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:42858/x_/rw"}
[junit4:junit4]   2> 634146 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:42828_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:42828/x_/rw"}
[junit4:junit4]   2> 634159 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634160 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634160 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634160 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634160 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634160 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634159 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634159 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 634806 T3044 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node5, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 634806 T3044 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node5&state=recovering&nodeName=127.0.0.1:42858_x_%252Frw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 635001 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:33030_x_%2Frw
[junit4:junit4]   2> 635003 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:33030_x_%2Frw
[junit4:junit4]   2> 635004 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 635005 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 635005 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 6)
[junit4:junit4]   2> 635005 T3118 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635006 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635006 T3100 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635005 T3021 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635007 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635007 T3067 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635007 T3050 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635006 T3014 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635008 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635008 T3034 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635007 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635008 T3083 oascc.ZkStateReader$3.process Updating live nodes... (7)
[junit4:junit4]   2> 635008 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635013 T3119 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 635013 T3119 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 635014 T3119 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 635666 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 635667 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node6",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:33030_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:33030/x_/rw"}
[junit4:junit4]   2> 635667 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 635667 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard3
[junit4:junit4]   2> 635716 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 635716 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 636014 T3119 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 636015 T3119 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1
[junit4:junit4]   2> 636015 T3119 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 636016 T3119 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 636016 T3119 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 636017 T3119 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/'
[junit4:junit4]   2> 636018 T3119 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/lib/README' to classloader
[junit4:junit4]   2> 636018 T3119 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 636056 T3119 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 636096 T3119 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 636097 T3119 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 636101 T3119 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 636552 T3119 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 636558 T3119 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 636561 T3119 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 636571 T3119 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 636574 T3119 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 636577 T3119 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 636578 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 636578 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 636578 T3119 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 636579 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 636579 T3119 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 636579 T3119 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 636579 T3119 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty6-1372413457471/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/
[junit4:junit4]   2> 636580 T3119 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 636580 T3119 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6
[junit4:junit4]   2> 636581 T3119 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index/
[junit4:junit4]   2> 636581 T3119 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 636581 T3119 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index
[junit4:junit4]   2> 636593 T3119 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty6/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 636593 T3119 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 636595 T3119 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 636595 T3119 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 636596 T3119 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 636596 T3119 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 636597 T3119 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 636597 T3119 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 636597 T3119 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 636598 T3119 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 636598 T3119 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 636601 T3119 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 636605 T3119 oass.SolrIndexSearcher.<init> Opening Searcher@727a6 main
[junit4:junit4]   2> 636606 T3119 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 636606 T3119 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 636612 T3120 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@727a6 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 636614 T3119 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 636614 T3119 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:33030/x_/rw collection:collection1 shard:shard3
[junit4:junit4]   2> 636618 T3119 oasc.ZkController.register We are http://127.0.0.1:33030/x_/rw/collection1/ and leader is http://127.0.0.1:40386/x_/rw/collection1/
[junit4:junit4]   2> 636618 T3119 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:33030/x_/rw
[junit4:junit4]   2> 636618 T3119 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 636618 T3119 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C497 name=collection1 org.apache.solr.core.SolrCore@11e2f19 url=http://127.0.0.1:33030/x_/rw/collection1 node=127.0.0.1:33030_x_%2Frw C497_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:33030_x_%2Frw, base_url=http://127.0.0.1:33030/x_/rw}
[junit4:junit4]   2> 636619 T3121 C497 P33030 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 636622 T3121 C497 P33030 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 636622 T3119 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 636622 T3121 C497 P33030 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 636623 T3121 C497 P33030 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 636623 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 636624 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 636624 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 636626 T3121 C497 P33030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 636630 T3063 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 636691 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 636694 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47110
[junit4:junit4]   2> 636695 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 636695 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 636695 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295
[junit4:junit4]   2> 636696 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/solr.xml
[junit4:junit4]   2> 636696 T2992 oasc.CoreContainer.<init> New CoreContainer 13795504
[junit4:junit4]   2> 636697 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/'
[junit4:junit4]   2> 636697 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/'
[junit4:junit4]   2> 636788 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 636789 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 636789 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 636790 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 636790 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 636791 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 636791 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 636792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 636792 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 636792 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 636797 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 636797 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 636798 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 636799 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 636800 T3133 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8280b name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 636802 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 636804 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> ASYNC  NEW_CORE C498 name=collection1 org.apache.solr.core.SolrCore@1ffb77f url=http://127.0.0.1:42858/x_/rw/collection1 node=127.0.0.1:42858_x_%2Frw C498_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:42858_x_%2Frw, base_url=http://127.0.0.1:42858/x_/rw}
[junit4:junit4]   2> 636807 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40786/x_/rw/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 636807 T3103 C498 P42858 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:42858/x_/rw START replicas=[http://127.0.0.1:40786/x_/rw/collection1/] nUpdates=100
[junit4:junit4]   2> 636808 T3103 C498 P42858 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 636808 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 636808 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 636808 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 636809 T3103 C498 P42858 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 636809 T3103 C498 P42858 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40786/x_/rw/collection1/. core=collection1
[junit4:junit4]   2> 636809 T3103 C498 P42858 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 636814 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 636816 T3136 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ea8ed1 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 636816 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 636820 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C499 name=collection1 org.apache.solr.core.SolrCore@92cef2 url=http://127.0.0.1:40786/x_/rw/collection1 node=127.0.0.1:40786_x_%2Frw C499_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40786_x_%2Frw, base_url=http://127.0.0.1:40786/x_/rw, leader=true}
[junit4:junit4]   2> 636821 T3044 C499 P40786 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=2
[junit4:junit4]   2> 636824 T3043 C499 P40786 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 636827 T3043 C499 P40786 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty2/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 636827 T3043 C499 P40786 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 636828 T3043 C499 P40786 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 636828 T3043 C499 P40786 oass.SolrIndexSearcher.<init> Opening Searcher@1adc008 realtime
[junit4:junit4]   2> 636829 T3043 C499 P40786 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 636829 T3043 C499 P40786 oasup.LogUpdateProcessor.finish [collection1] webapp=/x_/rw path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 5
[junit4:junit4]   2> 636830 T3103 C498 P42858 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 636830 T3103 C498 P42858 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 636832 T3045 C499 P40786 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 636833 T3045 C499 P40786 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=2
[junit4:junit4]   2> 636833 T3103 C498 P42858 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 636833 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 636834 T3103 C498 P42858 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 636834 T3103 C498 P42858 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 636835 T3103 C498 P42858 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 637222 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 637222 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node6",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard3",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:33030_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:33030/x_/rw"}
[junit4:junit4]   2> 637225 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node5",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":"shard2",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:42858_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:42858/x_/rw"}
[junit4:junit4]   2> 637229 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637230 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637631 T3063 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node6, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 637631 T3063 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node6&state=recovering&nodeName=127.0.0.1:33030_x_%252Frw&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 637824 T2992 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47110_x_%2Frw
[junit4:junit4]   2> 637825 T2992 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47110_x_%2Frw
[junit4:junit4]   2> 637826 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637827 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637827 T3021 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637827 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 637826 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 7)
[junit4:junit4]   2> 637827 T3118 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637827 T3100 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637828 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 637828 T3050 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637829 T3034 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637828 T3136 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637828 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 637830 T3014 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637830 T3136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 637829 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 637830 T3067 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637830 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 637831 T3083 oascc.ZkStateReader$3.process Updating live nodes... (8)
[junit4:junit4]   2> 637839 T3137 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 637839 T3137 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 637840 T3137 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 638739 T3015 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 638740 T3015 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=3 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node7",
[junit4:junit4]   2>  "numShards":"3",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:47110_x_%2Frw",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:47110/x_/rw"}
[junit4:junit4]   2> 638740 T3015 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=3
[junit4:junit4]   2> 638740 T3015 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 638796 T3050 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3034 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3136 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3100 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3067 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3118 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3021 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638796 T3014 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 8)
[junit4:junit4]   2> 638840 T3137 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 638841 T3137 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1
[junit4:junit4]   2> 638841 T3137 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 638842 T3137 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 638842 T3137 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 638843 T3137 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/'
[junit4:junit4]   2> 638844 T3137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/lib/README' to classloader
[junit4:junit4]   2> 638844 T3137 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 638881 T3137 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 638929 T3137 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 638930 T3137 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 638934 T3137 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 639364 T3137 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 639370 T3137 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 639372 T3137 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 639383 T3137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 639386 T3137 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 639389 T3137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 639390 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 639390 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 639390 T3137 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 639391 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 639391 T3137 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 639392 T3137 oasc.SolrCore.initDirectoryFactory solr.StandardDirectoryFactory
[junit4:junit4]   2> 639392 T3137 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty7-1372413460295/collection1/, dataDir=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/
[junit4:junit4]   2> 639392 T3137 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@ff291e
[junit4:junit4]   2> 639393 T3137 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7
[junit4:junit4]   2> 639393 T3137 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index/
[junit4:junit4]   2> 639393 T3137 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 639394 T3137 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index
[junit4:junit4]   2> 639397 T3137 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty7/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 639397 T3137 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 639399 T3137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 639399 T3137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 639400 T3137 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 639400 T3137 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 639401 T3137 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 639401 T3137 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 639402 T3137 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 639402 T3137 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 639403 T3137 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 639405 T3137 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 639408 T3137 oass.SolrIndexSearcher.<init> Opening Searcher@18a2663 main
[junit4:junit4]   2> 639409 T3137 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 639409 T3137 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 639416 T3138 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@18a2663 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 639419 T3137 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 639419 T3137 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47110/x_/rw collection:collection1 shard:shard1
[junit4:junit4]   2> 639421 T3137 oasc.ZkController.register We are http://127.0.0.1:47110/x_/rw/collection1/ and leader is http://127.0.0.1:33049/x_/rw/collection1/
[junit4:junit4]   2> 639421 T3137 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47110/x_/rw
[junit4:junit4]   2> 639421 T3137 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 639421 T3137 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C500 name=collection1 org.apache.solr.core.SolrCore@6d95cf url=http://127.0.0.1:47110/x_/rw/collection1 node=127.0.0.1:47110_x_%2Frw C500_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:47110_x_%2Frw, base_url=http://127.0.0.1:47110/x_/rw}
[junit4:junit4]   2> 639422 T3139 C500 P47110 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 639422 T3137 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 639425 T3139 C500 P47110 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 639425 T3139 C500 P47110 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 639426 T3139 C500 P47110 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 639426 T2992 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1
[junit4:junit4]   2> 639426 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 639427 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 639429 T3139 C500 P47110 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 639434 T3030 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node7, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 639499 T2992 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 639501 T2992 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:33646
[junit4:junit4]   2> 639502 T2992 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 639502 T2992 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 639502 T2992 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098
[junit4:junit4]   2> 639503 T2992 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098/solr.xml
[junit4:junit4]   2> 639503 T2992 oasc.CoreContainer.<init> New CoreContainer 2223439
[junit4:junit4]   2> 639503 T2992 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098/'
[junit4:junit4]   2> 639504 T2992 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-jetty8-1372413463098/'
[junit4:junit4]   2> 639578 T2992 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 639578 T2992 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 639579 T2992 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 639579 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 639579 T2992 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 639580 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 639580 T2992 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 639580 T2992 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 639581 T2992 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 639581 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 639584 T2992 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 639585 T2992 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:51741/solr
[junit4:junit4]   2> 639585 T2992 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 639586 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 639587 T3151 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@94c5a4 name:ZooKeeperConnection Watcher:127.0.0.1:51741 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 639588 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 639589 T2992 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 639593 T2992 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 639594 T3153 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1375e23 name:ZooKeeperConnection Watcher:127.0.0.1:51741/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 639594 T2992 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 639596 T2992 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C501 name=collection1 org.apache.solr.core.SolrCore@11e2f19 url=http://127.0.0.1:33030/x_/rw/collection1 node=127.0.0.1:33030_x_%2Frw C501_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:33030_x_%2Frw, base_url=http://127.0.0.1:33030/x_/rw}
[junit4:junit4]   2> 639632 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:40386/x_/rw/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 639633 T3121 C501 P33030 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:33030/x_/rw START replicas=[http://127.0.0.1:40386/x_/rw/collection1/] nUpdates=100
[junit4:junit4]   2> 639633 T3121 C501 P33030 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 639634 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 639634 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 639634 T3121 C501 P33030 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 639634 T3121 C501 P33030 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 639635 T3121 C501 P33030 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:40386/x_/rw/collection1/. core=collection1
[junit4:junit4]   2> 639635 T3121 C501 P33030 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C502 name=collection1 org.apache.solr.core.SolrCore@e58d0c url=http://127.0.0.1:40386/x_/rw/collection1 node=127.0.0.1:40386_x_%2Frw C502_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:40386_x_%2Frw, base_url=http://127.0.0.1:40386/x_/rw, leader=true}
[junit4:junit4]   2> 639639 T3062 C502 P40386 oasc.SolrCore.execute [collection1] webapp=/x_/rw path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 639639 T3060 C502 P40386 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 639641 T3060 C502 P40386 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.ChaosMonkeySafeLeaderTest-1372413435935/jetty3/index,segFN=segments_1,generation=1}
[junit4:junit4]   2> 639642 T3060 C502 P40386 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 639642 T3060 C502 P40386 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit

[...truncated too long message...]

39 T2992 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 45695
[junit4:junit4]   2> 691377 T2992 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
[junit4:junit4]   2> 691378 T2992 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:51741 51741
[junit4:junit4]   1> client port:0.0.0.0/0.0.0.0:0
[junit4:junit4]   1> safe stop:
[junit4:junit4]   1> safe stop:
[junit4:junit4]   2> 691382 T2992 oas.SolrTestCaseJ4.deleteCore ###deleteCore
[junit4:junit4]   2> 79147 T2991 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 4 leaked thread(s).
[junit4:junit4]   2> 691458 T3207 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 691458 T3207 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 691459 T3207 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 99287 T2991 ccr.ThreadLeakControl.checkThreadLeaks SEVERE 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
[junit4:junit4]   2>   1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4]   2>        at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]   2>        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4]   2>        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4]   2>        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4]   2>        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4]   2>        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4]   2>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4]   2>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2>        at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> 99287 T2991 ccr.ThreadLeakControl.tryToInterruptAll Starting to interrupt leaked threads:
[junit4:junit4]   2>   1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4]   2> 102289 T2991 ccr.ThreadLeakControl.tryToInterruptAll SEVERE There are still zombie threads that couldn't be terminated:
[junit4:junit4]   2>   1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4]   2>        at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]   2>        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4]   2>        at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4]   2>        at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4]   2>        at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4]   2>        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4]   2>        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4]   2>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]   2>        at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]   2> NOTE: test params are: codec=Lucene42: {range_facet_l=PostingsFormat(name=MockRandom), text=PostingsFormat(name=Direct), _version_=PostingsFormat(name=MockRandom), rnd_b=Pulsing41(freqCutoff=5 minBlockSize=53 maxBlockSize=114), intDefault=PostingsFormat(name=MockRandom), timestamp=PostingsFormat(name=MockRandom), id=Pulsing41(freqCutoff=5 minBlockSize=53 maxBlockSize=114), a_t=PostingsFormat(name=MockRandom), range_facet_sl=PostingsFormat(name=Direct), range_facet_si=PostingsFormat(name=NestedPulsing), other_tl1=PostingsFormat(name=MockRandom), multiDefault=PostingsFormat(name=NestedPulsing), a_si=PostingsFormat(name=NestedPulsing)}, docValues:{timestamp=DocValuesFormat(name=CheapBastard)}, sim=DefaultSimilarity, locale=es_PA, timezone=America/Chicago
[junit4:junit4]   2> NOTE: Linux 3.2.0-48-generic i386/Oracle Corporation 1.7.0_25 (32-bit)/cpus=8,threads=1,free=167126544,total=383561728
[junit4:junit4]   2> NOTE: All tests run in this JVM: [OpenCloseCoreStressTest, RequestHandlersTest, TestReload, TestJmxMonitoredMap, TestRecovery, FullSolrCloudDistribCmdsTest, TestPropInject, SampleTest, FieldAnalysisRequestHandlerTest, SpellCheckComponentTest, TestIndexingPerformance, TestWordDelimiterFilterFactory, LeaderElectionIntegrationTest, TestFoldingMultitermQuery, TestUniqueKeyFieldResource, TestSchemaResource, TestJmxIntegration, TestRecoveryHdfs, TermVectorComponentDistributedTest, DOMUtilTest, TestAnalyzedSuggestions, EchoParamsTest, TestCloudManagedSchema, HighlighterTest, TestRTGBase, BasicDistributedZk2Test, TestStressLucene, SpellPossibilityIteratorTest, ZkNodePropsTest, TestLMJelinekMercerSimilarityFactory, ShardRoutingCustomTest, TestUpdate, SimplePostToolTest, TestDefaultSimilarityFactory, TestSolrJ, DebugComponentTest, CachingDirectoryFactoryTest, TestMultiCoreConfBootstrap, ConvertedLegacyTest, PingRequestHandlerTest, ShardSplitTest, UpdateParamsTest, UniqFieldsUpdateProcessorFactoryTest, TestIndexSearcher, TestSolrXMLSerializer, TestBinaryField, HdfsBasicDistributedZk2Test, TestCoreDiscovery, ChaosMonkeySafeLeaderTest]
[junit4:junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=ChaosMonkeySafeLeaderTest -Dtests.seed=7BB20C2B3A27495C -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es_PA -Dtests.timezone=America/Chicago -Dtests.file.encoding=UTF-8
[junit4:junit4] ERROR   0.00s J1 | ChaosMonkeySafeLeaderTest (suite) <<<
[junit4:junit4]    > Throwable #1: com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE scope at org.apache.solr.cloud.ChaosMonkeySafeLeaderTest:
[junit4:junit4]    >    1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4]    >         at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)Throwable #2: com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie threads that couldn't be terminated:
[junit4:junit4]    >    1) Thread[id=3052, name=IPC Parameter Sending Thread #3, state=TIMED_WAITING, group=TGRP-TestRecoveryHdfs]
[junit4:junit4]    >         at sun.misc.Unsafe.park(Native Method)
[junit4:junit4]    >         at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
[junit4:junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
[junit4:junit4]    >         at java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:359)
[junit4:junit4]    >         at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:942)
[junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
[junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
[junit4:junit4]    >         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4]    >         at java.lang.Thread.run(Thread.java:724)
[junit4:junit4]    > at __randomizedtesting.SeedInfo.seed([7BB20C2B3A27495C]:0)
[junit4:junit4] Completed on J1 in 102.37s, 1 test, 2 errors <<< FAILURES!

[...truncated 563 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:386: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:366: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build.xml:190: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/common-build.xml:443: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:1245: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/lucene/common-build.xml:889: There were test failures: 312 suites, 1316 tests, 2 suite-level errors, 694 ignored (6 assumptions)

Total time: 31 minutes 23 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.7.0_25 -client -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.7.0_25) - Build # 6349 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/6349/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
REGRESSION:  org.apache.solr.cloud.SyncSliceTest.testDistribSearch

Error Message:
shard1 is not consistent.  Got 305 from http://127.0.0.1:45675/uw_b/cf/collection1lastClient and got 261 from http://127.0.0.1:44231/uw_b/cf/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 305 from http://127.0.0.1:45675/uw_b/cf/collection1lastClient and got 261 from http://127.0.0.1:44231/uw_b/cf/collection1
        at __randomizedtesting.SeedInfo.seed([4C71556C638378C7:CD97DB7414DC18FB]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1045)
        at org.apache.solr.cloud.SyncSliceTest.doTest(SyncSliceTest.java:237)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:724)




Build Log:
[...truncated 9268 lines...]
[junit4:junit4] Suite: org.apache.solr.cloud.SyncSliceTest
[junit4:junit4]   2> 57167 T231 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /uw_b/cf
[junit4:junit4]   2> 57175 T231 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
[junit4:junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./solrtest-SyncSliceTest-1372418129605
[junit4:junit4]   2> 57176 T231 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4:junit4]   2> 57177 T232 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
[junit4:junit4]   2> 57277 T231 oasc.ZkTestServer.run start zk server on port:53479
[junit4:junit4]   2> 57278 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 57283 T238 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45ea5f99 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 57283 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 57284 T231 oascc.SolrZkClient.makePath makePath: /solr
[junit4:junit4]   2> 57288 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 57289 T240 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@5d476c50 name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 57290 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 57290 T231 oascc.SolrZkClient.makePath makePath: /collections/collection1
[junit4:junit4]   2> 57294 T231 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
[junit4:junit4]   2> 57299 T231 oascc.SolrZkClient.makePath makePath: /collections/control_collection
[junit4:junit4]   2> 57303 T231 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
[junit4:junit4]   2> 57306 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 57307 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
[junit4:junit4]   2> 57312 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
[junit4:junit4]   2> 57313 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
[junit4:junit4]   2> 57317 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
[junit4:junit4]   2> 57318 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
[junit4:junit4]   2> 57321 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
[junit4:junit4]   2> 57322 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
[junit4:junit4]   2> 57325 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
[junit4:junit4]   2> 57326 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
[junit4:junit4]   2> 57330 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 57330 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
[junit4:junit4]   2> 57334 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 57334 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
[junit4:junit4]   2> 57338 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 57339 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
[junit4:junit4]   2> 57342 T231 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
[junit4:junit4]   2> 57343 T231 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
[junit4:junit4]   2> 57434 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 57438 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:49202
[junit4:junit4]   2> 57438 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 57439 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 57439 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799
[junit4:junit4]   2> 57439 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/solr.xml
[junit4:junit4]   2> 57440 T231 oasc.CoreContainer.<init> New CoreContainer 1792687692
[junit4:junit4]   2> 57440 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/'
[junit4:junit4]   2> 57440 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/'
[junit4:junit4]   2> 57485 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 57486 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 57486 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 57487 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 57487 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 57487 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 57488 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 57488 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 57488 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 57489 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 57493 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 57494 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4]   2> 57494 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 57495 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 57497 T251 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f0e8369 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 57497 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 57499 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 57506 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 57509 T253 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6efe55ac name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 57509 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 57512 T231 oascc.SolrZkClient.makePath makePath: /overseer/queue
[junit4:junit4]   2> 57517 T231 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
[junit4:junit4]   2> 57521 T231 oascc.SolrZkClient.makePath makePath: /live_nodes
[junit4:junit4]   2> 57523 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:49202_uw_b%2Fcf
[junit4:junit4]   2> 57525 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:49202_uw_b%2Fcf
[junit4:junit4]   2> 57529 T231 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
[junit4:junit4]   2> 57535 T231 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
[junit4:junit4]   2> 57540 T231 oasc.Overseer.start Overseer (id=89942794542055427-127.0.0.1:49202_uw_b%2Fcf-n_0000000000) starting
[junit4:junit4]   2> 57544 T231 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
[junit4:junit4]   2> 57549 T255 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
[junit4:junit4]   2> 57549 T231 oascc.SolrZkClient.makePath makePath: /clusterstate.json
[junit4:junit4]   2> 57552 T231 oascc.SolrZkClient.makePath makePath: /aliases.json
[junit4:junit4]   2> 57554 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 57557 T254 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
[junit4:junit4]   2> 57560 T256 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 57560 T256 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 57561 T256 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 59063 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 59064 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:49202_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:49202/uw_b/cf"}
[junit4:junit4]   2> 59064 T254 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with numShards 1
[junit4:junit4]   2> 59065 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 59071 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 59562 T256 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 59562 T256 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1
[junit4:junit4]   2> 59562 T256 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
[junit4:junit4]   2> 59564 T256 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 59564 T256 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
[junit4:junit4]   2> 59566 T256 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/'
[junit4:junit4]   2> 59566 T256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/lib/README' to classloader
[junit4:junit4]   2> 59567 T256 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 59594 T256 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 59639 T256 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 59641 T256 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 59650 T256 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 60032 T256 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 60053 T256 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 60058 T256 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 60081 T256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 60085 T256 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 60089 T256 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 60091 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 60091 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 60092 T256 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 60093 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 60093 T256 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 60094 T256 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 60094 T256 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-controljetty-1372418129799/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/
[junit4:junit4]   2> 60094 T256 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4]   2> 60095 T256 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data
[junit4:junit4]   2> 60095 T256 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/index/
[junit4:junit4]   2> 60095 T256 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 60096 T256 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/control/data/index
[junit4:junit4]   2> 60097 T256 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_1,generation=1}
[junit4:junit4]   2> 60098 T256 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 60100 T256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 60100 T256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 60101 T256 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 60101 T256 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 60102 T256 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 60102 T256 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 60102 T256 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 60103 T256 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 60103 T256 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 60105 T256 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 60108 T256 oass.SolrIndexSearcher.<init> Opening Searcher@10279ee2 main
[junit4:junit4]   2> 60109 T256 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 60109 T256 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 60113 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@10279ee2 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 60115 T256 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 60116 T256 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:49202/uw_b/cf collection:control_collection shard:shard1
[junit4:junit4]   2> 60117 T256 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
[junit4:junit4]   2> 60126 T256 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 60129 T256 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 60129 T256 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 60129 T256 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:49202/uw_b/cf/collection1/
[junit4:junit4]   2> 60130 T256 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 60130 T256 oasc.SyncStrategy.syncToMe http://127.0.0.1:49202/uw_b/cf/collection1/ has no replicas
[junit4:junit4]   2> 60130 T256 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:49202/uw_b/cf/collection1/ shard1
[junit4:junit4]   2> 60130 T256 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
[junit4:junit4]   2> 60577 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 60587 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
[junit4:junit4]   2> 60638 T256 oasc.ZkController.register We are http://127.0.0.1:49202/uw_b/cf/collection1/ and leader is http://127.0.0.1:49202/uw_b/cf/collection1/
[junit4:junit4]   2> 60638 T256 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:49202/uw_b/cf
[junit4:junit4]   2> 60638 T256 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 60639 T256 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 60639 T256 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 60641 T256 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 60642 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 60643 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 60643 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 60647 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:
[junit4:junit4]   2> 60650 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 60651 T260 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@785d564b name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 60652 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 60653 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 60657 T231 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
[junit4:junit4]   2> 60721 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 60723 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45675
[junit4:junit4]   2> 60724 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 60725 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 60725 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087
[junit4:junit4]   2> 60725 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/solr.xml
[junit4:junit4]   2> 60726 T231 oasc.CoreContainer.<init> New CoreContainer 2095146885
[junit4:junit4]   2> 60726 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/'
[junit4:junit4]   2> 60726 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/'
[junit4:junit4]   2> 60770 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 60771 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 60771 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 60772 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 60772 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 60772 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 60773 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 60773 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 60773 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 60774 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 60777 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 60778 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4]   2> 60778 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 60779 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 60782 T271 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@73918082 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 60782 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 60784 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 60789 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 60790 T273 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7b480442 name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 60791 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 60796 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 61801 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45675_uw_b%2Fcf
[junit4:junit4]   2> 61802 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45675_uw_b%2Fcf
[junit4:junit4]   2> 61806 T260 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 61807 T253 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 61807 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 61807 T273 oascc.ZkStateReader$3.process Updating live nodes... (2)
[junit4:junit4]   2> 61814 T274 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 61815 T274 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 61817 T274 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 62094 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 62095 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"control_collection",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:49202_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:49202/uw_b/cf"}
[junit4:junit4]   2> 62100 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4]   2> 62100 T254 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with numShards 1
[junit4:junit4]   2> 62100 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 62105 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 62106 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 62106 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 62818 T274 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 62818 T274 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1
[junit4:junit4]   2> 62818 T274 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 62819 T274 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 62820 T274 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 62821 T274 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/'
[junit4:junit4]   2> 62822 T274 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/lib/README' to classloader
[junit4:junit4]   2> 62822 T274 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 62849 T274 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 62910 T274 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 62916 T274 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 62924 T274 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 63311 T274 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 63317 T274 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 63319 T274 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 63348 T274 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 63353 T274 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 63359 T274 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 63361 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 63361 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 63362 T274 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 63363 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 63364 T274 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 63364 T274 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 63365 T274 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty1-1372418133087/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/
[junit4:junit4]   2> 63365 T274 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4]   2> 63366 T274 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1
[junit4:junit4]   2> 63366 T274 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index/
[junit4:junit4]   2> 63367 T274 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 63367 T274 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index
[junit4:junit4]   2> 63370 T274 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4]   2> 63370 T274 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 63374 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 63374 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 63375 T274 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 63376 T274 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 63377 T274 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 63377 T274 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 63378 T274 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 63379 T274 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 63379 T274 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 63383 T274 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 63387 T274 oass.SolrIndexSearcher.<init> Opening Searcher@223e4532 main
[junit4:junit4]   2> 63388 T274 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 63388 T274 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 63395 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@223e4532 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 63398 T274 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 63399 T274 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:45675/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4]   2> 63400 T274 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
[junit4:junit4]   2> 63410 T274 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 63413 T274 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
[junit4:junit4]   2> 63414 T274 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 63414 T274 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 63415 T274 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 63415 T274 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/ has no replicas
[junit4:junit4]   2> 63415 T274 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45675/uw_b/cf/collection1/ shard1
[junit4:junit4]   2> 63416 T274 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 63614 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 63623 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 63623 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 63623 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 63675 T274 oasc.ZkController.register We are http://127.0.0.1:45675/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 63675 T274 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:45675/uw_b/cf
[junit4:junit4]   2> 63676 T274 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
[junit4:junit4]   2> 63676 T274 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 63676 T274 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 63679 T274 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 63681 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 63681 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 63682 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 63749 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 63751 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44900
[junit4:junit4]   2> 63752 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 63752 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 63753 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115
[junit4:junit4]   2> 63753 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/solr.xml
[junit4:junit4]   2> 63753 T231 oasc.CoreContainer.<init> New CoreContainer 226056211
[junit4:junit4]   2> 63754 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/'
[junit4:junit4]   2> 63754 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/'
[junit4:junit4]   2> 63808 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 63809 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 63809 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 63810 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 63810 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 63810 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 63811 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 63811 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 63811 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 63812 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 63815 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 63816 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4]   2> 63816 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 63817 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 63819 T287 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@76c86b31 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 63819 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 63821 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 63825 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 63826 T289 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@50021d3 name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 63826 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 63831 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 64834 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44900_uw_b%2Fcf
[junit4:junit4]   2> 64836 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44900_uw_b%2Fcf
[junit4:junit4]   2> 64839 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
[junit4:junit4]   2> 64839 T253 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 64840 T260 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 64840 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 64840 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 64840 T289 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 64842 T273 oascc.ZkStateReader$3.process Updating live nodes... (3)
[junit4:junit4]   2> 64846 T290 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 64846 T290 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 64848 T290 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 65131 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 65132 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4]   2> 65136 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4]   2> 65137 T254 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 65137 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 65142 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 65142 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 65142 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 65142 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 65848 T290 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 65849 T290 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1
[junit4:junit4]   2> 65849 T290 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 65850 T290 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 65850 T290 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 65852 T290 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/'
[junit4:junit4]   2> 65852 T290 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/lib/README' to classloader
[junit4:junit4]   2> 65853 T290 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 65879 T290 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 65909 T290 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 65910 T290 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 65916 T290 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 66350 T290 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 66358 T290 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 66360 T290 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 66385 T290 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 66389 T290 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 66394 T290 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 66395 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 66396 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 66396 T290 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 66398 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 66398 T290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 66398 T290 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 66398 T290 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty2-1372418136115/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/
[junit4:junit4]   2> 66399 T290 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4]   2> 66400 T290 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2
[junit4:junit4]   2> 66400 T290 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/index/
[junit4:junit4]   2> 66400 T290 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 66401 T290 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty2/index
[junit4:junit4]   2> 66402 T290 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_1,generation=1}
[junit4:junit4]   2> 66402 T290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 66405 T290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 66405 T290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 66406 T290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 66407 T290 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 66408 T290 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 66409 T290 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 66409 T290 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 66410 T290 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 66411 T290 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 66415 T290 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 66418 T290 oass.SolrIndexSearcher.<init> Opening Searcher@24803429 main
[junit4:junit4]   2> 66419 T290 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 66420 T290 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 66428 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@24803429 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 66431 T290 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 66432 T290 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44900/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4]   2> 66435 T290 oasc.ZkController.register We are http://127.0.0.1:44900/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 66436 T290 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44900/uw_b/cf
[junit4:junit4]   2> 66436 T290 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 66436 T290 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C21 name=collection1 org.apache.solr.core.SolrCore@7c20a9de url=http://127.0.0.1:44900/uw_b/cf/collection1 node=127.0.0.1:44900_uw_b%2Fcf C21_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf}
[junit4:junit4]   2> 66440 T292 C21 P44900 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 66441 T290 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 66441 T292 C21 P44900 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 66442 T292 C21 P44900 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 66442 T292 C21 P44900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 66443 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 66443 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 66444 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 66444 T292 C21 P44900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 66454 T267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 66550 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 66553 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44231
[junit4:junit4]   2> 66553 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 66554 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 66554 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879
[junit4:junit4]   2> 66554 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/solr.xml
[junit4:junit4]   2> 66555 T231 oasc.CoreContainer.<init> New CoreContainer 445244580
[junit4:junit4]   2> 66555 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/'
[junit4:junit4]   2> 66556 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/'
[junit4:junit4]   2> 66635 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 66635 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 66636 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 66637 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 66637 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 66637 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 66638 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 66642 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 66642 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4]   2> 66643 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 66643 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 66645 T304 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b16555b name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 66646 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 66647 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 66650 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 66651 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4]   2> 66652 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 66654 T306 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@632099ee name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 66656 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 66657 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 66657 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 66657 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 66657 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 66663 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> 67455 T267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node2, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 67456 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node2&state=recovering&nodeName=127.0.0.1:44900_uw_b%252Fcf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4]   2> 67667 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44231_uw_b%2Fcf
[junit4:junit4]   2> 67670 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44231_uw_b%2Fcf
[junit4:junit4]   2> 67673 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
[junit4:junit4]   2> 67674 T260 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 67674 T253 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 67674 T289 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 67674 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 67674 T306 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 67674 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 67674 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 67677 T273 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 67681 T307 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 67681 T307 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 67683 T307 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 68165 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 68166 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4]   2> 68167 T254 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 68167 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 68172 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 68172 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 68172 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 68172 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 68172 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 68683 T307 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 68684 T307 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1
[junit4:junit4]   2> 68684 T307 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 68685 T307 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 68685 T307 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 68687 T307 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/'
[junit4:junit4]   2> 68688 T307 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/lib/README' to classloader
[junit4:junit4]   2> 68688 T307 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 68717 T307 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 68750 T307 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 68752 T307 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 68758 T307 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 69131 T307 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 69137 T307 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 69139 T307 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 69160 T307 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 69167 T307 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 69170 T307 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 69172 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 69173 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 69173 T307 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 69175 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 69175 T307 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 69175 T307 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 69176 T307 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty3-1372418138879/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/
[junit4:junit4]   2> 69176 T307 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4]   2> 69177 T307 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3
[junit4:junit4]   2> 69177 T307 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/index/
[junit4:junit4]   2> 69177 T307 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 69177 T307 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty3/index
[junit4:junit4]   2> 69179 T307 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_1,generation=1}
[junit4:junit4]   2> 69179 T307 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 69181 T307 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 69181 T307 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 69182 T307 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 69183 T307 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 69183 T307 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 69183 T307 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 69184 T307 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 69184 T307 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 69185 T307 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 69187 T307 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 69190 T307 oass.SolrIndexSearcher.<init> Opening Searcher@3540be79 main
[junit4:junit4]   2> 69190 T307 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 69190 T307 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 69195 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3540be79 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 69198 T307 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 69198 T307 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44231/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4]   2> 69202 T307 oasc.ZkController.register We are http://127.0.0.1:44231/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 69202 T307 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44231/uw_b/cf
[junit4:junit4]   2> 69203 T307 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 69203 T307 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C22 name=collection1 org.apache.solr.core.SolrCore@4b2ea730 url=http://127.0.0.1:44231/uw_b/cf/collection1 node=127.0.0.1:44231_uw_b%2Fcf C22_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4]   2> 69203 T309 C22 P44231 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 69204 T307 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 69204 T309 C22 P44231 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 69204 T309 C22 P44231 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 69205 T309 C22 P44231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 69205 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 69206 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 69206 T309 C22 P44231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 69206 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 69212 T267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> 69274 T231 oejs.Server.doStart jetty-8.1.10.v20130312
[junit4:junit4]   2> 69276 T231 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44218
[junit4:junit4]   2> 69277 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
[junit4:junit4]   2> 69277 T231 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
[junit4:junit4]   2> 69277 T231 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639
[junit4:junit4]   2> 69278 T231 oasc.CoreContainer$Initializer.initialize looking for solr config file: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/solr.xml
[junit4:junit4]   2> 69278 T231 oasc.CoreContainer.<init> New CoreContainer 1190966907
[junit4:junit4]   2> 69278 T231 oasc.CoreContainer.load Loading CoreContainer using Solr Home: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/'
[junit4:junit4]   2> 69279 T231 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/'
[junit4:junit4]   2> 69334 T231 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
[junit4:junit4]   2> 69334 T231 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
[junit4:junit4]   2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
[junit4:junit4]   2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
[junit4:junit4]   2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
[junit4:junit4]   2> 69335 T231 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
[junit4:junit4]   2> 69336 T231 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
[junit4:junit4]   2> 69336 T231 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
[junit4:junit4]   2> 69336 T231 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
[junit4:junit4]   2> 69337 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
[junit4:junit4]   2> 69341 T231 oasl.LogWatcher.newRegisteredLogWatcher Registering Log Listener
[junit4:junit4]   2> 69341 T231 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53479/solr
[junit4:junit4]   2> 69342 T231 oasc.ZkController.checkChrootPath zkHost includes chroot
[junit4:junit4]   2> 69342 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 69344 T321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@39bd5f67 name:ZooKeeperConnection Watcher:127.0.0.1:53479 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 69344 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 69346 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
[junit4:junit4]   2> 69350 T231 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
[junit4:junit4]   2> 69351 T323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2c1548bb name:ZooKeeperConnection Watcher:127.0.0.1:53479/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
[junit4:junit4]   2> 69352 T231 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
[junit4:junit4]   2> 69355 T231 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C23 name=collection1 org.apache.solr.core.SolrCore@7c20a9de url=http://127.0.0.1:44900/uw_b/cf/collection1 node=127.0.0.1:44900_uw_b%2Fcf C23_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf}
[junit4:junit4]   2> 69457 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45675/uw_b/cf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 69457 T292 C23 P44900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 69458 T292 C23 P44900 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 69458 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 69458 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 69459 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 69459 T292 C23 P44900 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 69459 T292 C23 P44900 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45675/uw_b/cf/collection1/. core=collection1
[junit4:junit4]   2> 69459 T292 C23 P44900 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C24 name=collection1 org.apache.solr.core.SolrCore@33ff417c url=http://127.0.0.1:45675/uw_b/cf/collection1 node=127.0.0.1:45675_uw_b%2Fcf C24_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4]   2> 69464 T266 C24 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 69471 T268 C24 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 69473 T268 C24 P45675 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4]   2> 69473 T268 C24 P45675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 69473 T268 C24 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 69474 T268 C24 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@37c4d046 realtime
[junit4:junit4]   2> 69474 T268 C24 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 69474 T268 C24 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 69490 T292 C23 P44900 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 69491 T292 C23 P44900 oascsi.HttpClientUtil.createClient Creating new http client, config:connTimeout=5000&socketTimeout=20000&allowCompression=false&maxConnections=10000&maxConnectionsPerHost=10000
[junit4:junit4]   2> 69496 T292 C23 P44900 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 69499 T266 C24 P45675 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 69499 T266 C24 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=1
[junit4:junit4]   2> 69500 T292 C23 P44900 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 69500 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 69501 T292 C23 P44900 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 69501 T292 C23 P44900 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 69502 T292 C23 P44900 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 69679 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 69681 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4]   2> 69685 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node2",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4]   2> 69690 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 69690 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 69691 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 69690 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 69690 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 69691 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 70212 T267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 70213 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node3&state=recovering&nodeName=127.0.0.1:44231_uw_b%252Fcf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1001
[junit4:junit4]   2> 70359 T231 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44218_uw_b%2Fcf
[junit4:junit4]   2> 70361 T231 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44218_uw_b%2Fcf
[junit4:junit4]   2> 70363 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 70363 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 70363 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 70363 T260 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 70364 T289 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 70364 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 70364 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 70366 T253 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 70366 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 70366 T323 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 70368 T306 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 70368 T273 oascc.ZkStateReader$3.process Updating live nodes... (5)
[junit4:junit4]   2> 70372 T325 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 70372 T325 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 70374 T325 oasc.ZkController.waitForCoreNodeName look for our core node name
[junit4:junit4]   2> 71200 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 71201 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4]   2> 71201 T254 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=1
[junit4:junit4]   2> 71202 T254 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
[junit4:junit4]   2> 71206 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 71207 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 71207 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 71207 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 71207 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 71207 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 71374 T325 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
[junit4:junit4]   2> 71374 T325 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1
[junit4:junit4]   2> 71375 T325 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
[junit4:junit4]   2> 71375 T325 oasc.ZkController.createCollectionZkNode Collection zkNode exists
[junit4:junit4]   2> 71376 T325 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
[junit4:junit4]   2> 71377 T325 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/'
[junit4:junit4]   2> 71378 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/lib/README' to classloader
[junit4:junit4]   2> 71378 T325 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/lib/classes/' to classloader
[junit4:junit4]   2> 71410 T325 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_50
[junit4:junit4]   2> 71452 T325 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
[junit4:junit4]   2> 71454 T325 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
[junit4:junit4]   2> 71458 T325 oass.IndexSchema.readSchema [collection1] Schema name=test
[junit4:junit4]   2> 71840 T325 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
[junit4:junit4]   2> 71847 T325 oass.IndexSchema.readSchema default search field in schema is text
[junit4:junit4]   2> 71849 T325 oass.IndexSchema.readSchema unique key field: id
[junit4:junit4]   2> 71870 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 71875 T325 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
[junit4:junit4]   2> 71881 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 71882 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 71883 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 71883 T325 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
[junit4:junit4]   2> 71884 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
[junit4:junit4]   2> 71885 T325 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
[junit4:junit4]   2> 71885 T325 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
[junit4:junit4]   2> 71885 T325 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.SyncSliceTest-jetty4-1372418141639/collection1/, dataDir=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/
[junit4:junit4]   2> 71885 T325 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@150858bb
[junit4:junit4]   2> 71886 T325 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4
[junit4:junit4]   2> 71886 T325 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/index/
[junit4:junit4]   2> 71887 T325 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/index' doesn't exist. Creating new index...
[junit4:junit4]   2> 71887 T325 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty4/index
[junit4:junit4]   2> 71889 T325 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_1,generation=1}
[junit4:junit4]   2> 71889 T325 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 71892 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
[junit4:junit4]   2> 71892 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
[junit4:junit4]   2> 71892 T325 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
[junit4:junit4]   2> 71893 T325 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
[junit4:junit4]   2> 71894 T325 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
[junit4:junit4]   2> 71894 T325 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
[junit4:junit4]   2> 71894 T325 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
[junit4:junit4]   2> 71895 T325 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
[junit4:junit4]   2> 71896 T325 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
[junit4:junit4]   2> 71898 T325 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
[junit4:junit4]   2> 71900 T325 oass.SolrIndexSearcher.<init> Opening Searcher@4863c568 main
[junit4:junit4]   2> 71901 T325 oasu.CommitTracker.<init> Hard AutoCommit: disabled
[junit4:junit4]   2> 71901 T325 oasu.CommitTracker.<init> Soft AutoCommit: disabled
[junit4:junit4]   2> 71909 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4863c568 main{StandardDirectoryReader(segments_1:1)}
[junit4:junit4]   2> 71912 T325 oasc.CoreContainer.registerCore registering core: collection1
[junit4:junit4]   2> 71912 T325 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:44218/uw_b/cf collection:collection1 shard:shard1
[junit4:junit4]   2> 71917 T325 oasc.ZkController.register We are http://127.0.0.1:44218/uw_b/cf/collection1/ and leader is http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 71917 T325 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:44218/uw_b/cf
[junit4:junit4]   2> 71917 T325 oasc.ZkController.checkRecovery Core needs to recover:collection1
[junit4:junit4]   2> 71918 T325 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
[junit4:junit4]   2> ASYNC  NEW_CORE C25 name=collection1 org.apache.solr.core.SolrCore@345c4417 url=http://127.0.0.1:44218/uw_b/cf/collection1 node=127.0.0.1:44218_uw_b%2Fcf C25_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:44218_uw_b%2Fcf, base_url=http://127.0.0.1:44218/uw_b/cf}
[junit4:junit4]   2> 71918 T327 C25 P44218 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
[junit4:junit4]   2> 71919 T327 C25 P44218 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
[junit4:junit4]   2> 71919 T325 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 71919 T327 C25 P44218 oasc.ZkController.publish publishing core=collection1 state=recovering
[junit4:junit4]   2> 71920 T327 C25 P44218 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 71921 T231 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0
[junit4:junit4]   2> 71921 T231 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
[junit4:junit4]   2> 71921 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 71923 T327 C25 P44218 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 71926 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 71928 T231 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 30 for each attempt
[junit4:junit4]   2> 71928 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):30
[junit4:junit4]   2> 71930 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 71930 T267 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true
[junit4:junit4]   2> ASYNC  NEW_CORE C26 name=collection1 org.apache.solr.core.SolrCore@4b2ea730 url=http://127.0.0.1:44231/uw_b/cf/collection1 node=127.0.0.1:44231_uw_b%2Fcf C26_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4]   2> 72214 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45675/uw_b/cf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 72215 T309 C26 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 72215 T309 C26 P44231 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 72216 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 72216 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 72216 T309 C26 P44231 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 72217 T309 C26 P44231 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 72217 T309 C26 P44231 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45675/uw_b/cf/collection1/. core=collection1
[junit4:junit4]   2> 72217 T309 C26 P44231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> ASYNC  NEW_CORE C27 name=collection1 org.apache.solr.core.SolrCore@33ff417c url=http://127.0.0.1:45675/uw_b/cf/collection1 node=127.0.0.1:45675_uw_b%2Fcf C27_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4]   2> 72218 T266 C27 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> ASYNC  NEW_CORE C28 name=collection1 org.apache.solr.core.SolrCore@33ff417c url=http://127.0.0.1:45675/uw_b/cf/collection1 node=127.0.0.1:45675_uw_b%2Fcf C28_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4]   2> 72237 T269 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 72237 T269 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 72238 T269 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 72238 T269 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 2
[junit4:junit4]   2> ASYNC  NEW_CORE C29 name=collection1 org.apache.solr.core.SolrCore@4b2ea730 url=http://127.0.0.1:44231/uw_b/cf/collection1 node=127.0.0.1:44231_uw_b%2Fcf C29_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4]   2> 72239 T309 C29 P44231 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 72240 T309 C29 P44231 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 72243 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 72244 T309 C29 P44231 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 72244 T309 C29 P44231 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 72244 T309 C29 P44231 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 72245 T309 C29 P44231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 72246 T309 C29 P44231 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 72714 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 72716 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"recovering",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4]   2> 72721 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node3",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44231_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4]   2> 72726 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 72726 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 72727 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 72726 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 72726 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 72726 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 72931 T267 oasha.CoreAdminHandler.handleWaitForStateAction Waited coreNodeName: core_node4, state: recovering, checkLive: true, onlyIfLeader: true for: 1 seconds.
[junit4:junit4]   2> 72932 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={coreNodeName=core_node4&state=recovering&nodeName=127.0.0.1:44218_uw_b%252Fcf&action=PREPRECOVERY&checkLive=true&core=collection1&wt=javabin&onlyIfLeader=true&version=2} status=0 QTime=1002
[junit4:junit4]   2> 72932 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 73934 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> ASYNC  NEW_CORE C30 name=collection1 org.apache.solr.core.SolrCore@345c4417 url=http://127.0.0.1:44218/uw_b/cf/collection1 node=127.0.0.1:44218_uw_b%2Fcf C30_STATE=coll:collection1 core:collection1 props:{state=recovering, core=collection1, node_name=127.0.0.1:44218_uw_b%2Fcf, base_url=http://127.0.0.1:44218/uw_b/cf}
[junit4:junit4]   2> 74933 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Attempting to PeerSync from http://127.0.0.1:45675/uw_b/cf/collection1/ core=collection1 - recoveringAfterStartup=true
[junit4:junit4]   2> 74933 T327 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 74933 T327 C30 P44218 oasu.PeerSync.sync WARN no frame of reference to tell of we've missed updates
[junit4:junit4]   2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery PeerSync Recovery was not successful - trying replication. core=collection1
[junit4:junit4]   2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Starting Replication Recovery. core=collection1
[junit4:junit4]   2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Begin buffering updates. core=collection1
[junit4:junit4]   2> 74934 T327 C30 P44218 oasu.UpdateLog.bufferUpdates Starting to buffer updates. FSUpdateLog{state=ACTIVE, tlog=null}
[junit4:junit4]   2> 74934 T327 C30 P44218 oasc.RecoveryStrategy.replicate Attempting to replicate from http://127.0.0.1:45675/uw_b/cf/collection1/. core=collection1
[junit4:junit4]   2> 74935 T327 C30 P44218 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 74936 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 74939 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 74941 T266 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=false,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 74942 T266 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 74942 T266 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 74942 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&openSearcher=false&commit=true&wt=javabin&commit_end_point=true&version=2&softCommit=false} {commit=} 0 1
[junit4:junit4]   2> 74943 T327 C30 P44218 oash.ReplicationHandler.inform Commits will be reserved for  10000
[junit4:junit4]   2> 74943 T327 C30 P44218 oash.SnapPuller.<init>  No value set for 'pollInterval'. Timer Task not started.
[junit4:junit4]   2> 74946 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/replication params={command=indexversion&qt=/replication&wt=javabin&version=2} status=0 QTime=0
[junit4:junit4]   2> 74947 T327 C30 P44218 oasc.RecoveryStrategy.replay No replay needed. core=collection1
[junit4:junit4]   2> 74947 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Replication Recovery was successful - registering as Active. core=collection1
[junit4:junit4]   2> 74947 T327 C30 P44218 oasc.ZkController.publish publishing core=collection1 state=active
[junit4:junit4]   2> 74947 T327 C30 P44218 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 74948 T327 C30 P44218 oasc.RecoveryStrategy.doRecovery Finished recovery process. core=collection1
[junit4:junit4]   2> 75739 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 75740 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node4",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"active",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:44218_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4]   2> 75746 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 75746 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 75746 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 75746 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 75746 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 75746 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 75942 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 75943 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> ASYNC  NEW_CORE C31 name=collection1 org.apache.solr.core.SolrCore@667376df url=http://127.0.0.1:49202/uw_b/cf/collection1 node=127.0.0.1:49202_uw_b%2Fcf C31_STATE=coll:control_collection core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:49202_uw_b%2Fcf, base_url=http://127.0.0.1:49202/uw_b/cf, leader=true}
[junit4:junit4]   2> 75951 T249 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 75953 T249 C31 P49202 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_1,generation=1}
[junit4:junit4]   2> 75954 T249 C31 P49202 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 75955 T249 C31 P49202 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 75955 T249 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@46bdc822 main
[junit4:junit4]   2> 75956 T249 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 75957 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@46bdc822 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 75957 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> ASYNC  NEW_CORE C32 name=collection1 org.apache.solr.core.SolrCore@7c20a9de url=http://127.0.0.1:44900/uw_b/cf/collection1 node=127.0.0.1:44900_uw_b%2Fcf C32_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf}
[junit4:junit4]   2> 75968 T285 C32 P44900 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:45675/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 75979 T282 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 75983 T268 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 75984 T268 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2>  C29_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44231_uw_b%2Fcf, base_url=http://127.0.0.1:44231/uw_b/cf}
[junit4:junit4]   2> 75984 T301 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 75986 T268 C28 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@668911a6 main
[junit4:junit4]   2>  C30_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44218_uw_b%2Fcf, base_url=http://127.0.0.1:44218/uw_b/cf}
[junit4:junit4]   2> 75987 T316 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 75988 T301 C29 P44231 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_1,generation=1}
[junit4:junit4]   2> 75986 T282 C32 P44900 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_1,generation=1}
[junit4:junit4]   2> 75988 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@668911a6 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 75987 T268 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 75989 T316 C30 P44218 oasc.SolrDeletionPolicy.onInit SolrDeletionPolicy.onInit: commits: num=1
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_1,generation=1}
[junit4:junit4]   2> 75989 T282 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 75989 T301 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 75992 T282 C32 P44900 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 75992 T316 C30 P44218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
[junit4:junit4]   2> 75991 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 8
[junit4:junit4]   2> 76002 T316 C30 P44218 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 76008 T301 C29 P44231 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 76009 T282 C32 P44900 oass.SolrIndexSearcher.<init> Opening Searcher@17f7e334 main
[junit4:junit4]   2> 76011 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@17f7e334 main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 76012 T282 C32 P44900 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 76013 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4]   2> 76014 T316 C30 P44218 oass.SolrIndexSearcher.<init> Opening Searcher@5a0ed63f main
[junit4:junit4]   2> 76016 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a0ed63f main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 76017 T316 C30 P44218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 76017 T316 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 31
[junit4:junit4]   2> 76019 T301 C29 P44231 oass.SolrIndexSearcher.<init> Opening Searcher@39048c5d main
[junit4:junit4]   2> 76019 T301 C29 P44231 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 76020 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@39048c5d main{StandardDirectoryReader(segments_1:1:nrt)}
[junit4:junit4]   2> 76021 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 37
[junit4:junit4]   2> 76022 T285 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 54
[junit4:junit4]   2> 76024 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 76028 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 76032 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4]   2> 76036 T300 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=0
[junit4:junit4]   2> 76040 T318 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=0 status=0 QTime=1
[junit4:junit4]   2> 78051 T247 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {deleteByQuery=*:* (-1439084734552473600)} 0 7
[junit4:junit4]   2> 78072 T300 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=FROMLEADER&_version_=-1439084734566105088&update.from=http://127.0.0.1:45675/uw_b/cf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 1
[junit4:junit4]   2> 78072 T316 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=FROMLEADER&_version_=-1439084734566105088&update.from=http://127.0.0.1:45675/uw_b/cf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 1
[junit4:junit4]   2> 78085 T285 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=FROMLEADER&_version_=-1439084734566105088&update.from=http://127.0.0.1:45675/uw_b/cf/collection1/&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 2
[junit4:junit4]   2> 78086 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={update.distrib=TOLEADER&wt=javabin&version=2} {deleteByQuery=*:* (-1439084734566105088)} 0 27
[junit4:junit4]   2> 78087 T318 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {deleteByQuery=*:*} 0 33
[junit4:junit4]   2> 78100 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[0 (1439084734600708096)]} 0 9
[junit4:junit4]   2> 78121 T319 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 3
[junit4:junit4]   2> 78121 T284 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 3
[junit4:junit4]   2> 78124 T302 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 4
[junit4:junit4]   2> 78125 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:44231/uw_b/cf/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[0 (1439084734620631040)]} 0 13
[junit4:junit4]   2> 78126 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[0]} 0 22
[junit4:junit4]   2> 78131 T248 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[1 (1439084734640553984)]} 0 1
[junit4:junit4]   2> 78151 T317 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 4
[junit4:junit4]   2> 78154 T283 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 8
[junit4:junit4]   2> 78156 T299 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 6
[junit4:junit4]   2> 78158 T267 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:44231/uw_b/cf/collection1/&update.distrib=TOLEADER&wt=javabin&version=2} {add=[1 (1439084734651039744)]} 0 18
[junit4:junit4]   2> 78159 T300 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[1]} 0 24
[junit4:junit4]   2> 78164 T246 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[2 (1439084734675156992)]} 0 2
[junit4:junit4]   2> 78168 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44900/uw_b/cf/collection1/]
[junit4:junit4]   2> 78169 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44900/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/] result:true
[junit4:junit4]   2> 78169 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44231/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/] result:false
[junit4:junit4]   2> 78169 T268 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44218/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/] result:false
[junit4:junit4]   2> 78178 T316 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[2 (1439084734681448448)]} 0 4
[junit4:junit4]   2> 78180 T302 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[2 (1439084734681448448)]} 0 4
[junit4:junit4]   2> 78181 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[2 (1439084734681448448)]} 0 13
[junit4:junit4]   2> 78187 T247 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[3 (1439084734698225664)]} 0 2
[junit4:junit4]   2> 78192 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/]
[junit4:junit4]   2> 78193 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44900/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/] result:true
[junit4:junit4]   2> 78193 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44231/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/] result:true
[junit4:junit4]   2> 78194 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44218/uw_b/cf/collection1/ against:[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/] result:false
[junit4:junit4]   2> 78201 T318 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&wt=javabin&version=2} {add=[3 (1439084734707662848)]} 0 2
[junit4:junit4]   2> 78203 T269 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={test.distrib.skip.servers=http://127.0.0.1:44900/uw_b/cf/collection1/&test.distrib.skip.servers=http://127.0.0.1:44231/uw_b/cf/collection1/&wt=javabin&version=2} {add=[3 (1439084734707662848)]} 0 12
[junit4:junit4]   2> 78206 T249 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78221 T249 C31 P49202 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_2,generation=2}
[junit4:junit4]   2> 78222 T249 C31 P49202 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 78230 T249 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@3b5125e8 main
[junit4:junit4]   2> 78231 T249 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78232 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b5125e8 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 78233 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4]   2> 78237 T266 C28 P45675 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:45675/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 78241 T301 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78241 T282 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78241 T267 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78242 T319 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78256 T301 C29 P44231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_2,generation=2}
[junit4:junit4]   2> 78256 T282 C32 P44900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_2,generation=2}
[junit4:junit4]   2> 78257 T267 C28 P45675 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_2,generation=2}
[junit4:junit4]   2> 78259 T267 C28 P45675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 78256 T301 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 78258 T282 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 78268 T267 C28 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@474a1d0 main
[junit4:junit4]   2> 78268 T319 C30 P44218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2211c4af lockFactory=org.apache.lucene.store.NativeFSLockFactory@240c8343),segFN=segments_2,generation=2}
[junit4:junit4]   2> 78272 T267 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78272 T319 C30 P44218 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 2
[junit4:junit4]   2> 78274 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@474a1d0 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 78275 T267 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 34
[junit4:junit4]   2> 78276 T282 C32 P44900 oass.SolrIndexSearcher.<init> Opening Searcher@79382710 main
[junit4:junit4]   2> 78276 T319 C30 P44218 oass.SolrIndexSearcher.<init> Opening Searcher@2d153029 main
[junit4:junit4]   2> 78277 T282 C32 P44900 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78278 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@79382710 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C2)}
[junit4:junit4]   2> 78279 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 38
[junit4:junit4]   2> 78277 T301 C29 P44231 oass.SolrIndexSearcher.<init> Opening Searcher@338353fd main
[junit4:junit4]   2> 78277 T319 C30 P44218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78281 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@338353fd main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C3)}
[junit4:junit4]   2> 78280 T301 C29 P44231 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78280 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@2d153029 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 78284 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 44
[junit4:junit4]   2> 78285 T319 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 43
[junit4:junit4]   2> 78286 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 49
[junit4:junit4]   2> 78287 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):330
[junit4:junit4]   2> 78288 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 78289 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 78292 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 78296 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=2 status=0 QTime=0
[junit4:junit4]   2> 78300 T299 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=3 status=0 QTime=0
[junit4:junit4]   2> 78304 T317 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 78307 T231 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 78314 T300 oasha.CollectionsHandler.handleSyncShardAction Syncing shard : shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2
[junit4:junit4]   2> 78315 T300 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
[junit4:junit4]   2> 78323 T267 oasha.CoreAdminHandler.handleRequestSyncAction I have been requested to sync up my shard
[junit4:junit4]   2> 78323 T267 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 78324 T267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf START replicas=[http://127.0.0.1:44900/uw_b/cf/collection1/, http://127.0.0.1:44231/uw_b/cf/collection1/, http://127.0.0.1:44218/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 78328 T284 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 78328 T316 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 78328 T302 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 78329 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf  Received 3 versions from 127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4]   2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734651039744
[junit4:junit4]   2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf  Received 5 versions from 127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4]   2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734681448448
[junit4:junit4]   2> 78330 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf  Received 4 versions from 127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4]   2> 78331 T267 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734681448448
[junit4:junit4]   2> 78331 T267 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:45675/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 78331 T267 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 78331 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: try and ask http://127.0.0.1:44900/uw_b/cf/collection1/ to sync
[junit4:junit4]   2> 78332 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: try and ask http://127.0.0.1:44231/uw_b/cf/collection1/ to sync
[junit4:junit4]   2> 78333 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/: try and ask http://127.0.0.1:44218/uw_b/cf/collection1/ to sync
[junit4:junit4]   2> 78336 T283 C32 P44900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 78336 T299 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 78339 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf START replicas=[http://127.0.0.1:45675/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 78339 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 78340 T283 C32 P44900 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf  Received 5 versions from 127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 78341 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 78341 T283 C32 P44900 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf Requesting updates from 127.0.0.1:45675/uw_b/cf/collection1/n=2 versions=[1439084734707662848, 1439084734681448448]
[junit4:junit4]   2> 78342 T299 C29 P44231 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf  Received 5 versions from 127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 78342 T299 C29 P44231 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf Requesting updates from 127.0.0.1:45675/uw_b/cf/collection1/n=1 versions=[1439084734707662848]
[junit4:junit4]   2> 78343 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 78344 T318 C30 P44218 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf  Received 5 versions from 127.0.0.1:45675/uw_b/cf/collection1/
[junit4:junit4]   2> 78344 T318 C30 P44218 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734566105088 otherHigh=1439084734681448448
[junit4:junit4]   2> 78345 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 78345 T318 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:45675/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=6
[junit4:junit4]   2> 78346 T268 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={distrib=false&getUpdates=1439084734707662848,1439084734681448448&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 78346 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/:  sync completed with http://127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4]   2> 78348 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={distrib=false&getUpdates=1439084734707662848&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 78351 T299 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] {add=[3 (1439084734707662848)]} 0 2
[junit4:junit4]   2> 78352 T299 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 78352 T299 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:45675/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=16
[junit4:junit4]   2> 78353 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/:  sync completed with http://127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4]   2> 78354 T283 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] {add=[2 (1439084734681448448), 3 (1439084734707662848)]} 0 8
[junit4:junit4]   2> 78354 T283 C32 P44900 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 78355 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:45675/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=19
[junit4:junit4]   2> 78356 T267 oasc.SyncStrategy.syncToMe http://127.0.0.1:45675/uw_b/cf/collection1/:  sync completed with http://127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4]   2> 78356 T267 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/cores params={shard=shard1&action=REQUESTSYNCSHARD&core=collection1&collection=collection1&wt=javabin&version=2} status=0 QTime=33
[junit4:junit4]   2> 78357 T300 oass.SolrDispatchFilter.handleAdminRequest [admin] webapp=null path=/admin/collections params={shard=shard1&action=SYNCSHARD&collection=collection1&wt=javabin&version=2} status=0 QTime=45
[junit4:junit4]   2> 78358 T231 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 15 for each attempt
[junit4:junit4]   2> 78358 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):15
[junit4:junit4]   2> 78359 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 78361 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 78364 T246 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78365 T246 C31 P49202 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 78365 T246 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@3a5edc81 main
[junit4:junit4]   2> 78366 T246 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78368 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3a5edc81 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 78368 T246 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 4
[junit4:junit4]   2> 78372 T266 C28 P45675 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:45675/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2> 78376 T268 C28 P45675 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78376 T282 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78376 T319 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78376 T302 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 78377 T319 C30 P44218 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 78376 T268 C28 P45675 oasu.DirectUpdateHandler2.commit No uncommitted changes. Skipping IW.commit.
[junit4:junit4]   2> 78379 T268 C28 P45675 oass.SolrIndexSearcher.<init> Opening Searcher@4d5c9173 main
[junit4:junit4]   2> 78380 T268 C28 P45675 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78382 T275 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@4d5c9173 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 78379 T319 C30 P44218 oass.SolrIndexSearcher.<init> Opening Searcher@22be1af6 main
[junit4:junit4]   2> 78382 T268 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 6
[junit4:junit4]   2> 78383 T319 C30 P44218 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78383 T282 C32 P44900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_3,generation=3}
[junit4:junit4]   2> 78384 T282 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 78384 T326 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@22be1af6 main{StandardDirectoryReader(segments_2:3:nrt _0(5.0):C4)}
[junit4:junit4]   2> 78389 T319 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 13
[junit4:junit4]   2> 78390 T282 C32 P44900 oass.SolrIndexSearcher.<init> Opening Searcher@43b4e3fd main
[junit4:junit4]   2> 78391 T302 C29 P44231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_3,generation=3}
[junit4:junit4]   2> 78392 T302 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 78392 T291 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@43b4e3fd main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C2 _1(5.0):C2)}
[junit4:junit4]   2> 78394 T282 C32 P44900 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78394 T302 C29 P44231 oass.SolrIndexSearcher.<init> Opening Searcher@5a77cf58 main
[junit4:junit4]   2> 78395 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 19
[junit4:junit4]   2> 78396 T308 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@5a77cf58 main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C3 _1(5.0):C1)}
[junit4:junit4]   2> 78397 T302 C29 P44231 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 78398 T302 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&expungeDeletes=false&commit_end_point=true&version=2&softCommit=false} {commit=} 0 22
[junit4:junit4]   2> 78399 T266 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 27
[junit4:junit4]   2> 78400 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 78404 T269 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 78407 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 78410 T301 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2> 78414 T317 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=1
[junit4:junit4]   2> 80416 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> check const of shard1
[junit4:junit4]   2> client0
[junit4:junit4]   2> PROPS:core_node1:{"state":"active","core":"collection1","node_name":"127.0.0.1:45675_uw_b%2Fcf","base_url":"http://127.0.0.1:45675/uw_b/cf","leader":"true"}
[junit4:junit4]   2> 80419 T266 C28 P45675 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client1
[junit4:junit4]   2> PROPS:core_node2:{"state":"active","core":"collection1","node_name":"127.0.0.1:44900_uw_b%2Fcf","base_url":"http://127.0.0.1:44900/uw_b/cf"}
[junit4:junit4]   2> 80422 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client2
[junit4:junit4]   2> PROPS:core_node3:{"state":"active","core":"collection1","node_name":"127.0.0.1:44231_uw_b%2Fcf","base_url":"http://127.0.0.1:44231/uw_b/cf"}
[junit4:junit4]   2> 80425 T302 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> client3
[junit4:junit4]   2> PROPS:core_node4:{"state":"active","core":"collection1","node_name":"127.0.0.1:44218_uw_b%2Fcf","base_url":"http://127.0.0.1:44218/uw_b/cf"}
[junit4:junit4]   2> 80428 T319 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={tests=checkShardConsistency&q=*:*&distrib=false&wt=javabin&version=2&rows=0} hits=4 status=0 QTime=0
[junit4:junit4]   2>  live:true
[junit4:junit4]   2>  num:4
[junit4:junit4]   2>
[junit4:junit4]   2> 80431 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/select params={q=*:*&wt=javabin&version=2} hits=4 status=0 QTime=0
[junit4:junit4]   2> 80439 T249 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={wt=javabin&version=2} {add=[4 (1439084737058570240)]} 0 3
[junit4:junit4]   2> 80444 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest test.distrib.skip.servers was found and contains:[http://127.0.0.1:44218/uw_b/cf/collection1/]
[junit4:junit4]   2> 80445 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44900/uw_b/cf/collection1/ against:[http://127.0.0.1:44218/uw_b/cf/collection1/] result:false
[junit4:junit4]   2> 80445 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44231/uw_b/cf/collection1/ against:[http://127.0.0.1:44218/uw_b/cf/collection1/] result:false
[junit4:junit4]   2> 80445 T269 C28 P45675 oasup.DistributedUpdateProcessor.setupRequest check url:http://127.0.0.1:44218/uw_b/cf/collection1/ against:[http://127.0.0.1:44218/uw_b/cf/collection1/] result:true
[junit4:junit4]   2> 80452 T301 C29 P44231 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4 (1439084737069056000)]} 0 2
[junit4:junit4]   2> 80453 T282 C32 P44900 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:45675/uw_b/cf/collection1/&update.distrib=FROMLEADER&test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4 (1439084737069056000)]} 0 3
[junit4:junit4]   2> 80454 T269 C28 P45675 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={distrib.from=http://127.0.0.1:44218/uw_b/cf/collection1/&update.distrib=TOLEADER&test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4 (1439084737069056000)]} 0 10
[junit4:junit4]   2> 80454 T317 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={test.distrib.skip.servers=http://127.0.0.1:44218/uw_b/cf/collection1/&wt=javabin&version=2} {add=[4]} 0 12
[junit4:junit4]   2> 80455 T231 oasc.ChaosMonkey.monkeyLog monkey: kill shard! 45675
[junit4:junit4]   2> 80477 T231 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=2095146885
[junit4:junit4]   2> 81774 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 81776 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=null message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "shard":null,
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4]   2> 81776 T254 oasc.Overseer$ClusterStateUpdater.updateState shard=shard1 is already registered
[junit4:junit4]   2> 81780 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 81780 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 81780 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 81780 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 81780 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 81780 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 83480 T231 oasc.ZkController.publish publishing core=collection1 state=down
[junit4:junit4]   2> 83480 T231 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
[junit4:junit4]   2> 83482 T231 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@33ff417c
[junit4:junit4]   2> 83488 T231 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=6,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=1,adds=1,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=5,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
[junit4:junit4]   2> 83489 T231 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
[junit4:junit4]   2> 83489 T231 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
[junit4:junit4]   2> 83489 T231 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
[junit4:junit4]   2>  C28_STATE=coll:collection1 core:collection1 props:{state=down, core=collection1, node_name=127.0.0.1:45675_uw_b%2Fcf, base_url=http://127.0.0.1:45675/uw_b/cf, leader=true}
[junit4:junit4]   2> 83494 T231 C28 P45675 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=3
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_1,generation=1}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2179600a lockFactory=org.apache.lucene.store.NativeFSLockFactory@6cda313c),segFN=segments_3,generation=3}
[junit4:junit4]   2> 83495 T231 C28 P45675 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 83496 T231 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
[junit4:junit4]   2> 83497 T231 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
[junit4:junit4]   2> 83497 T231 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1 [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1;done=false>>]
[junit4:junit4]   2> 83497 T231 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1
[junit4:junit4]   2> 83498 T231 oasc.CachingDirectoryFactory.closeCacheValue looking to close ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index [CachedDir<<refCount=0;path=./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index;done=false>>]
[junit4:junit4]   2> 83498 T231 oasc.CachingDirectoryFactory.close Closing directory: ./org.apache.solr.cloud.SyncSliceTest-1372418129605/jetty1/index
[junit4:junit4]   2> 83500 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 83500 T273 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 83500 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 5)
[junit4:junit4]   2> 83501 T253 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 83501 T273 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 83502 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 83501 T231 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/uw_b/cf,null}
[junit4:junit4]   2> 83502 T273 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
[junit4:junit4]   2> 83504 T260 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 83505 T289 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 83505 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 83505 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 83509 T323 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 83509 T306 oascc.ZkStateReader$3.process Updating live nodes... (4)
[junit4:junit4]   2> 83512 T289 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
[junit4:junit4]   2> 83515 T289 oasc.ShardLeaderElectionContext.shouldIBeLeader Checking if I should try and be the leader.
[junit4:junit4]   2> 83515 T289 oasc.ShardLeaderElectionContext.shouldIBeLeader My last published State was Active, it's okay to be the leader.
[junit4:junit4]   2> 83515 T289 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
[junit4:junit4]   2> 83515 T289 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4]   2> 83515 T289 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf START replicas=[http://127.0.0.1:44231/uw_b/cf/collection1/, http://127.0.0.1:44218/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 83519 T319 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 83519 T302 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=0
[junit4:junit4]   2> 83520 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf  Received 5 versions from 127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4]   2> 83520 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734620631040 otherHigh=1439084734681448448
[junit4:junit4]   2> 83521 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf  Received 6 versions from 127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4]   2> 83521 T289 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734620631040 otherHigh=1439084734707662848
[junit4:junit4]   2> 83521 T289 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44900/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 83522 T289 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
[junit4:junit4]   2> 83522 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/: try and ask http://127.0.0.1:44231/uw_b/cf/collection1/ to sync
[junit4:junit4]   2> 83522 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/: try and ask http://127.0.0.1:44218/uw_b/cf/collection1/ to sync
[junit4:junit4]   2> 83525 T301 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf START replicas=[http://127.0.0.1:44900/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 83525 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf START replicas=[http://127.0.0.1:44900/uw_b/cf/collection1/] nUpdates=100
[junit4:junit4]   2> 83528 T284 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 83529 T318 C30 P44218 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf  Received 6 versions from 127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4]   2> 83529 T318 C30 P44218 oasu.PeerSync.requestUpdates PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf Requesting updates from 127.0.0.1:44900/uw_b/cf/collection1/n=1 versions=[1439084737069056000]
[junit4:junit4]   2> 83530 T285 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 83530 T301 C29 P44231 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf  Received 6 versions from 127.0.0.1:44900/uw_b/cf/collection1/
[junit4:junit4]   2> 83531 T301 C29 P44231 oasu.PeerSync.handleVersions PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf  Our versions are newer. ourLowThreshold=1439084734620631040 otherHigh=1439084734707662848
[junit4:junit4]   2> 83531 T301 C29 P44231 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44231/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 83532 T283 C32 P44900 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={distrib=false&getUpdates=1439084737069056000&wt=javabin&qt=/get&version=2} status=0 QTime=1
[junit4:junit4]   2> 83532 T301 C29 P44231 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:44900/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=7
[junit4:junit4]   2> 83533 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/:  sync completed with http://127.0.0.1:44231/uw_b/cf/collection1/
[junit4:junit4]   2> 83535 T318 C30 P44218 oasup.LogUpdateProcessor.finish [collection1] {add=[4 (1439084737069056000)]} 0 3
[junit4:junit4]   2> 83535 T318 C30 P44218 oasu.PeerSync.sync PeerSync: core=collection1 url=http://127.0.0.1:44218/uw_b/cf DONE. sync succeeded
[junit4:junit4]   2> 83536 T318 C30 P44218 oasc.SolrCore.execute [collection1] webapp=/uw_b/cf path=/get params={sync=http://127.0.0.1:44900/uw_b/cf/collection1/&getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=11
[junit4:junit4]   2> 83537 T289 oasc.SyncStrategy.syncToMe http://127.0.0.1:44900/uw_b/cf/collection1/:  sync completed with http://127.0.0.1:44218/uw_b/cf/collection1/
[junit4:junit4]   2> 83537 T289 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:44900/uw_b/cf/collection1/ shard1
[junit4:junit4]   2> 83537 T289 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
[junit4:junit4]   2> 83554 T231 oasc.ChaosMonkey.monkeyLog monkey: stop shard! 45675
[junit4:junit4]   2> 84789 T254 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 84790 T254 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
[junit4:junit4]   2>  "operation":"state",
[junit4:junit4]   2>  "core_node_name":"core_node1",
[junit4:junit4]   2>  "numShards":"1",
[junit4:junit4]   2>  "shard":"shard1",
[junit4:junit4]   2>  "roles":null,
[junit4:junit4]   2>  "state":"down",
[junit4:junit4]   2>  "shard_state":"active",
[junit4:junit4]   2>  "core":"collection1",
[junit4:junit4]   2>  "collection":"collection1",
[junit4:junit4]   2>  "shard_range":null,
[junit4:junit4]   2>  "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4]   2>  "base_url":"http://127.0.0.1:45675/uw_b/cf"}
[junit4:junit4]   2> 84808 T260 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84808 T289 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84808 T323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84808 T306 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 84808 T253 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
[junit4:junit4]   2> 85555 T231 oasc.AbstractFullDistribZkTestBase.waitForThingsToLevelOut Wait for recoveries to finish - wait 90 for each attempt
[junit4:junit4]   2> 85555 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Wait for recoveries to finish - collection: collection1 failOnTimeout:true timeout (sec):90
[junit4:junit4]   2> 85556 T231 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
[junit4:junit4]   2> 85558 T231 oasc.AbstractDistribZkTestBase.waitForRecoveriesToFinish Recoveries finished - collection: collection1
[junit4:junit4]   2> 85560 T247 C31 P49202 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 85564 T247 C31 P49202 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_2,generation=2}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@211782bf lockFactory=org.apache.lucene.store.NativeFSLockFactory@8504643),segFN=segments_3,generation=3}
[junit4:junit4]   2> 85565 T247 C31 P49202 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 3
[junit4:junit4]   2> 85566 T247 C31 P49202 oass.SolrIndexSearcher.<init> Opening Searcher@579abafe main
[junit4:junit4]   2> 85567 T247 C31 P49202 oasu.DirectUpdateHandler2.commit end_commit_flush
[junit4:junit4]   2> 85569 T257 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@579abafe main{StandardDirectoryReader(segments_3:5:nrt _0(5.0):C4 _1(5.0):C1)}
[junit4:junit4]   2> 85569 T247 C31 P49202 oasup.LogUpdateProcessor.finish [collection1] webapp=/uw_b/cf path=/update params={waitSearcher=true&commit=true&wt=javabin&version=2&softCommit=false} {commit=} 0 9
[junit4:junit4]   2> 85573 T319 C30 P44218 oasu.SolrCmdDistributor.distribCommit Distrib commit to:[StdNode: http://127.0.0.1:44900/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44231/uw_b/cf/collection1/, StdNode: http://127.0.0.1:44218/uw_b/cf/collection1/] params:commit_end_point=true&commit=true&softCommit=false&waitSearcher=true&expungeDeletes=false
[junit4:junit4]   2>  C32_STATE=coll:collection1 core:collection1 props:{state=active, core=collection1, node_name=127.0.0.1:44900_uw_b%2Fcf, base_url=http://127.0.0.1:44900/uw_b/cf, leader=true}
[junit4:junit4]   2> 85576 T285 C32 P44900 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 85576 T301 C29 P44231 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 85576 T316 C30 P44218 oasu.DirectUpdateHandler2.commit start commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
[junit4:junit4]   2> 85581 T301 C29 P44231 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_3,generation=3}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@3267b8c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@4389c038),segFN=segments_4,generation=4}
[junit4:junit4]   2> 85581 T285 C32 P44900 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_3,generation=3}
[junit4:junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@64f56312 lockFactory=org.apache.lucene.store.NativeFSLockFactory@50b7fbb8),segFN=segments_4,generation=4}
[junit4:junit4]   2> 85581 T285 C32 P44900 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 85581 T301 C29 P44231 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 4
[junit4:junit4]   2> 85582 T316 C30 P44218 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=2
[junit4:junit4]   2> commit{dir=MockDirW

[...truncated too long message...]

{
[junit4:junit4]   1>           "shards":{"shard1":{
[junit4:junit4]   1>               "range":"80000000-7fffffff",
[junit4:junit4]   1>               "state":"active",
[junit4:junit4]   1>               "replicas":{
[junit4:junit4]   1>                 "core_node1":{
[junit4:junit4]   1>                   "state":"active",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:45675_uw_b%2Fcf",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:45675/uw_b/cf"},
[junit4:junit4]   1>                 "core_node2":{
[junit4:junit4]   1>                   "state":"down",
[junit4:junit4]   1>                   "core":"collection1",
[junit4:junit4]   1>                   "node_name":"127.0.0.1:44900_uw_b%2Fcf",
[junit4:junit4]   1>                   "base_url":"http://127.0.0.1:44900/uw_b/cf"},
[junit4:junit4]   1>                 "core_node3":{
[junit4:junit4] &