[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 6772 - Failure!

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 6772 - Failure!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6772/
Java: 32bit/jdk1.6.0_45 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Server refused connection at: http://127.0.0.1:57661

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:57661
        at __randomizedtesting.SeedInfo.seed([878F5A354C3164F2:669D42D3B6E04CE]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:764)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:57661 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        ... 44 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 51 more




Build Log:
[...truncated 9251 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 653478 T2061 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 653482 T2061 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1375479197046
   [junit4]   2> 653482 T2061 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 653483 T2062 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 653583 T2061 oasc.ZkTestServer.run start zk server on port:53607
   [junit4]   2> 653584 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 653588 T2068 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@38a7b9 name:ZooKeeperConnection Watcher:127.0.0.1:53607 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 653588 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 653588 T2061 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 653593 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 653594 T2070 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@113fd7b name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 653595 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 653595 T2061 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 653599 T2061 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 653602 T2061 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 653610 T2061 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 653614 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 653614 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 653620 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 653620 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 653625 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 653625 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 653629 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 653629 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 653633 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 653633 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 653637 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 653637 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 653641 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 653641 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 653645 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 653645 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 653649 T2061 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 653649 T2061 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 653722 T2061 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 653724 T2061 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:53977
   [junit4]   2> 653725 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 653725 T2061 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 653726 T2061 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218
   [junit4]   2> 653726 T2061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/'
   [junit4]   2> 653738 T2061 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/solr.xml
   [junit4]   2> 653787 T2061 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 653788 T2061 oasc.CoreContainer.<init> New CoreContainer 1977916
   [junit4]   2> 653788 T2061 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/]
   [junit4]   2> 653789 T2061 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 653790 T2061 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 653790 T2061 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 653791 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 653791 T2061 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 653791 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 653792 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 653792 T2061 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 653792 T2061 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 653793 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 653796 T2061 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53607/solr
   [junit4]   2> 653797 T2061 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 653798 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 653799 T2081 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16d6d68 name:ZooKeeperConnection Watcher:127.0.0.1:53607 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 653799 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 653801 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 653805 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 653806 T2083 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f2375d name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 653806 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 653808 T2061 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 653813 T2061 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 653817 T2061 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 653820 T2061 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:53977_
   [junit4]   2> 653821 T2061 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:53977_
   [junit4]   2> 653827 T2061 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 653834 T2061 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 653837 T2061 oasc.Overseer.start Overseer (id=90143404657803267-127.0.0.1:53977_-n_0000000000) starting
   [junit4]   2> 653841 T2061 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 653845 T2085 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 653847 T2061 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 653851 T2061 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 653855 T2061 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 653859 T2084 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 653861 T2061 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218
   [junit4]   2> 653885 T2061 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/conf/
   [junit4]   2> 653886 T2061 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 653887 T2061 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 653887 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 653887 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 653890 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 653891 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 653893 T2088 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10d728f name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 653893 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 653895 T2061 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 653905 T2061 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 653974 T2061 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 653976 T2061 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57661
   [junit4]   2> 653977 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 653977 T2061 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 653977 T2061 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375479197470
   [junit4]   2> 653978 T2061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375479197470/'
   [junit4]   2> 653990 T2061 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375479197470/solr.xml
   [junit4]   2> 654038 T2061 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 654039 T2061 oasc.CoreContainer.<init> New CoreContainer 14528547
   [junit4]   2> 654039 T2061 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375479197470/]
   [junit4]   2> 654041 T2061 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 654041 T2061 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 654041 T2061 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 654042 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 654042 T2061 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 654042 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 654043 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 654043 T2061 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 654044 T2061 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 654044 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 654047 T2061 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53607/solr
   [junit4]   2> 654047 T2061 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 654048 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 654050 T2099 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@56fa7d name:ZooKeeperConnection Watcher:127.0.0.1:53607 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 654050 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 654052 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 654055 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 654056 T2101 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11e11ea name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 654056 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 654062 T2061 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 655066 T2061 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57661_
   [junit4]   2> 655067 T2061 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57661_
   [junit4]   2> 655072 T2101 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 655074 T2088 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 655074 T2083 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 655079 T2061 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375479197470
   [junit4]   2> 655103 T2061 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375479197470/conf/
   [junit4]   2> 655104 T2061 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 655105 T2061 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 655105 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 655106 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 655177 T2061 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 655179 T2061 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:36214
   [junit4]   2> 655180 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 655180 T2061 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 655180 T2061 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673
   [junit4]   2> 655181 T2061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/'
   [junit4]   2> 655203 T2061 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/solr.xml
   [junit4]   2> 655243 T2061 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 655244 T2061 oasc.CoreContainer.<init> New CoreContainer 11009522
   [junit4]   2> 655244 T2061 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/]
   [junit4]   2> 655245 T2061 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 655246 T2061 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 655246 T2061 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 655247 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 655247 T2061 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 655247 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 655248 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 655248 T2061 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 655248 T2061 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 655249 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 655252 T2061 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53607/solr
   [junit4]   2> 655252 T2061 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 655253 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 655255 T2113 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@11134d9 name:ZooKeeperConnection Watcher:127.0.0.1:53607 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 655255 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 655257 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 655260 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 655261 T2115 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@648a56 name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 655261 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 655267 T2061 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 656271 T2061 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:36214_
   [junit4]   2> 656274 T2061 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:36214_
   [junit4]   2> 656280 T2101 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 656281 T2088 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 656282 T2083 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 656283 T2115 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 656297 T2061 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673
   [junit4]   2> 656331 T2061 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/conf/
   [junit4]   2> 656331 T2061 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 656332 T2061 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 656332 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 656333 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 656428 T2061 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 656436 T2061 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38432
   [junit4]   2> 656437 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 656437 T2061 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 656438 T2061 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901
   [junit4]   2> 656438 T2061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/'
   [junit4]   2> 656461 T2061 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/solr.xml
   [junit4]   2> 656524 T2061 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 656524 T2061 oasc.CoreContainer.<init> New CoreContainer 2338168
   [junit4]   2> 656525 T2061 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/]
   [junit4]   2> 656527 T2061 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 656528 T2061 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 656528 T2061 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 656529 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 656529 T2061 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 656530 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 656530 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 656531 T2061 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 656531 T2061 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 656532 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 656536 T2061 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53607/solr
   [junit4]   2> 656536 T2061 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 656539 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 656540 T2127 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@51b4d7 name:ZooKeeperConnection Watcher:127.0.0.1:53607 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 656540 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 656543 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 656552 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 656555 T2129 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7022d4 name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 656559 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 656566 T2061 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 657578 T2061 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38432_
   [junit4]   2> 657582 T2061 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38432_
   [junit4]   2> 657590 T2083 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 657591 T2088 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 657592 T2129 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 657592 T2101 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 657592 T2115 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 657599 T2061 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901
   [junit4]   2> 657636 T2061 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/conf/
   [junit4]   2> 657636 T2061 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 657637 T2061 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 657637 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 657638 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 657750 T2061 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 657754 T2061 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55484
   [junit4]   2> 657755 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 657755 T2061 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 657755 T2061 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206
   [junit4]   2> 657756 T2061 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/'
   [junit4]   2> 657780 T2061 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/solr.xml
   [junit4]   2> 657841 T2061 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 657842 T2061 oasc.CoreContainer.<init> New CoreContainer 3691703
   [junit4]   2> 657842 T2061 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/]
   [junit4]   2> 657851 T2061 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 657852 T2061 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 657855 T2061 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 657859 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 657860 T2061 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 657860 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 657861 T2061 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 657861 T2061 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 657861 T2061 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 657862 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 657866 T2061 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:53607/solr
   [junit4]   2> 657866 T2061 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 657871 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 657872 T2141 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@da4b7 name:ZooKeeperConnection Watcher:127.0.0.1:53607 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 657872 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 657884 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 657891 T2061 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 657894 T2143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1415ba0 name:ZooKeeperConnection Watcher:127.0.0.1:53607/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 657894 T2061 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 657900 T2061 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 658914 T2061 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55484_
   [junit4]   2> 658916 T2061 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55484_
   [junit4]   2> 658925 T2088 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 658925 T2083 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 658925 T2129 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 658925 T2115 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 658926 T2101 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 658926 T2143 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 658933 T2061 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206
   [junit4]   2> 658963 T2061 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/conf/
   [junit4]   2> 658964 T2061 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 658964 T2061 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 658965 T2061 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 658965 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 658970 T2061 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 658975 T2095 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 658977 T2083 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658980 T2083 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 658980 T2085 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 658981 T2085 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 659874 T2084 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 659875 T2084 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: nodes_used_collection
   [junit4]   2> 659875 T2084 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with shards [shard1, shard2]
   [junit4]   2> 659880 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 659880 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 659880 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 659880 T2143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 659881 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 659881 T2129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 659986 T2085 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 659987 T2085 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:53977_
   [junit4]   2> 659987 T2085 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:38432_
   [junit4]   2> 659988 T2085 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:36214_
   [junit4]   2> 659989 T2085 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:55484_
   [junit4]   2> 659991 T2079 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
   [junit4]   2> 659992 T2139 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
   [junit4]   2> 659991 T2109 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
   [junit4]   2> 659991 T2123 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
   [junit4]   2> 659993 T2079 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 659995 T2109 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 659995 T2139 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 659995 T2123 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 661394 T2084 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 661396 T2084 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 661396 T2084 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:53977_",
   [junit4]   2>  "base_url":"http://127.0.0.1:53977"}
   [junit4]   2> 661404 T2084 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 661404 T2084 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node2",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:55484_",
   [junit4]   2>  "base_url":"http://127.0.0.1:55484"}
   [junit4]   2> 661412 T2084 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 661413 T2084 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node3",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:36214_",
   [junit4]   2>  "base_url":"http://127.0.0.1:36214"}
   [junit4]   2> 661420 T2084 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 661420 T2084 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node4",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:38432_",
   [junit4]   2>  "base_url":"http://127.0.0.1:38432"}
   [junit4]   2> 661426 T2101 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 661426 T2088 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 661426 T2143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 661427 T2115 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 661426 T2129 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 661426 T2083 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 661994 T2079 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica1
   [junit4]   2> 661996 T2123 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica2
   [junit4]   2> 661995 T2079 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/nodes_used_collection_shard1_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/nodes_used_collection_shard1_replica1/core.properties (No such file or directory)
   [junit4]   2> 661996 T2109 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica1
   [junit4]   2> 661996 T2139 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica2
   [junit4]   2> 661998 T2109 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/nodes_used_collection_shard2_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/nodes_used_collection_shard2_replica1/core.properties (No such file or directory)
   [junit4]   2> 661997 T2079 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/nodes_used_collection_shard1_replica1
   [junit4]   2> 661999 T2079 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 661997 T2123 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/nodes_used_collection_shard1_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/nodes_used_collection_shard1_replica2/core.properties (No such file or directory)
   [junit4]   2> 661998 T2109 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/nodes_used_collection_shard2_replica1
   [junit4]   2> 662000 T2109 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 661998 T2139 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/nodes_used_collection_shard2_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/nodes_used_collection_shard2_replica2/core.properties (No such file or directory)
   [junit4]   2> 662000 T2079 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 661999 T2123 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/nodes_used_collection_shard1_replica2
   [junit4]   2> 662001 T2123 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 662001 T2079 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 662000 T2139 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/nodes_used_collection_shard2_replica2
   [junit4]   2> 662010 T2139 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 662001 T2109 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 662010 T2123 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 662011 T2109 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 662012 T2123 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 662013 T2139 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 662014 T2139 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 662014 T2079 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 662014 T2079 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 662017 T2109 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 662017 T2123 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 662017 T2109 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 662017 T2123 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 662019 T2139 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 662019 T2139 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 662022 T2109 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 662024 T2123 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 662025 T2079 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 662026 T2109 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/nodes_used_collection_shard2_replica1/'
   [junit4]   2> 662026 T2139 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 662028 T2079 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/nodes_used_collection_shard1_replica1/'
   [junit4]   2> 662028 T2139 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/nodes_used_collection_shard2_replica2/'
   [junit4]   2> 662029 T2123 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/nodes_used_collection_shard1_replica2/'
   [junit4]   2> 662090 T2139 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 662093 T2109 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 662116 T2123 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 662118 T2079 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 662177 T2139 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 662179 T2109 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 662180 T2139 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 662181 T2109 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 662194 T2139 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
   [junit4]   2> 662198 T2109 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
   [junit4]   2> 662227 T2123 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 662229 T2079 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 662230 T2123 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 662231 T2079 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 662247 T2079 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
   [junit4]   2> 662247 T2123 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
   [junit4]   2> 663117 T2139 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663118 T2139 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663119 T2139 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 663135 T2139 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 663139 T2139 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 663271 T2139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663277 T2139 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663290 T2139 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663292 T2139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663293 T2139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663293 T2139 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663296 T2139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663296 T2139 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663297 T2139 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 663297 T2139 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//nodes_used_collection_shard2_replica2/data/
   [junit4]   2> 663298 T2139 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@904497
   [junit4]   2> 663299 T2139 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//nodes_used_collection_shard2_replica2/data
   [junit4]   2> 663300 T2139 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//nodes_used_collection_shard2_replica2/data/index/
   [junit4]   2> 663300 T2139 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 663301 T2139 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//nodes_used_collection_shard2_replica2/data/index
   [junit4]   2> 663310 T2139 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@197ba6 lockFactory=org.apache.lucene.store.NativeFSLockFactory@feafeb)),segFN=segments_1,generation=1}
   [junit4]   2> 663310 T2139 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 663313 T2079 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663314 T2079 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663314 T2079 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 663315 T2139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 663316 T2139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 663316 T2109 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663317 T2109 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663317 T2139 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 663318 T2109 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 663318 T2139 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 663320 T2139 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 663320 T2139 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 663320 T2139 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 663321 T2079 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 663321 T2139 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 663333 T2139 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 663336 T2139 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 663336 T2079 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 663338 T2109 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 663338 T2139 oass.SolrIndexSearcher.<init> Opening Searcher@6de6e1 main
   [junit4]   2> 663339 T2139 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 663339 T2139 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 663343 T2109 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 663345 T2123 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663346 T2123 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 663346 T2149 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@6de6e1 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 663346 T2139 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
   [junit4]   2> 663347 T2139 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:55484 collection:nodes_used_collection shard:shard2
   [junit4]   2> 663347 T2123 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 663347 T2139 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 663360 T2139 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
   [junit4]   2> 663362 T2123 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 663366 T2123 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 663380 T2079 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663386 T2109 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663388 T2079 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663390 T2139 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 663394 T2109 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663395 T2079 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663397 T2139 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=179999
   [junit4]   2> 663400 T2079 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663400 T2079 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663400 T2109 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663401 T2079 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663405 T2123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663405 T2079 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663405 T2109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663406 T2109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663405 T2079 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663406 T2109 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663406 T2079 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 663407 T2079 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218//nodes_used_collection_shard1_replica1/data/
   [junit4]   2> 663408 T2079 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@904497
   [junit4]   2> 663409 T2079 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218//nodes_used_collection_shard1_replica1/data
   [junit4]   2> 663409 T2079 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218//nodes_used_collection_shard1_replica1/data/index/
   [junit4]   2> 663410 T2079 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 663411 T2079 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375479197218//nodes_used_collection_shard1_replica1/data/index
   [junit4]   2> 663411 T2123 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 663411 T2109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663412 T2109 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663412 T2109 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 663413 T2079 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@1492d7a lockFactory=org.apache.lucene.store.NativeFSLockFactory@762e53)),segFN=segments_1,generation=1}
   [junit4]   2> 663413 T2109 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673//nodes_used_collection_shard2_replica1/data/
   [junit4]   2> 663413 T2079 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 663413 T2109 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@904497
   [junit4]   2> 663415 T2109 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673//nodes_used_collection_shard2_replica1/data
   [junit4]   2> 663415 T2109 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673//nodes_used_collection_shard2_replica1/data/index/
   [junit4]   2> 663415 T2109 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 663416 T2109 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375479198673//nodes_used_collection_shard2_replica1/data/index
   [junit4]   2> 663416 T2123 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663417 T2079 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 663418 T2079 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 663418 T2109 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@a525b4 lockFactory=org.apache.lucene.store.NativeFSLockFactory@11ffdf2)),segFN=segments_1,generation=1}
   [junit4]   2> 663418 T2109 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 663428 T2123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663427 T2079 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 663428 T2123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663429 T2123 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 663429 T2079 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 663430 T2079 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 663430 T2079 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 663431 T2079 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 663431 T2123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 663431 T2079 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 663431 T2123 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 663432 T2079 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 663432 T2123 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 663433 T2109 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 663433 T2123 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901//nodes_used_collection_shard1_replica2/data/
   [junit4]   2> 663433 T2109 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 663433 T2123 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@904497
   [junit4]   2> 663434 T2079 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 663434 T2109 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 663435 T2123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901//nodes_used_collection_shard1_replica2/data
   [junit4]   2> 663435 T2123 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901//nodes_used_collection_shard1_replica2/data/index/
   [junit4]   2> 663435 T2109 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 663435 T2123 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 663436 T2109 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 663436 T2123 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375479199901//nodes_used_collection_shard1_replica2/data/index
   [junit4]   2> 663436 T2109 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 663437 T2079 oass.SolrIndexSearcher.<init> Opening Searcher@17c747b main
   [junit4]   2> 663437 T2109 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 663438 T2079 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 663438 T2109 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 663438 T2079 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 663438 T2123 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(RateLimitedDirectoryWrapper(org.apache.lucene.store.RAMDirectory@334749 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1743a0b)),segFN=segments_1,generation=1}
   [junit4]   2> 663439 T2109 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 663439 T2123 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 663441 T2109 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 663442 T2150 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@17c747b main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 663443 T2123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 663443 T2079 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
   [junit4]   2> 663444 T2109 oass.SolrIndexSearcher.<init> Opening Searcher@8ff1ae main
   [junit4]   2> 663443 T2123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 663444 T2079 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:53977 collection:nodes_used_collection shard:shard1
   [junit4]   2> 663444 T2109 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 663445 T2079 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 663445 T2123 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 663445 T2109 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 663446 T2123 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 663447 T2123 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 663447 T2123 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 663447 T2123 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 663448 T2123 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 663448 T2123 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 663449 T2079 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 663449 T2151 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@8ff1ae main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 663451 T2123 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 663451 T2109 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
   [junit4]   2> 663452 T2109 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:36214 collection:nodes_used_collection shard:shard2
   [junit4]   2> 663452 T2109 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 663453 T2123 oass.SolrIndexSearcher.<init> Opening Searcher@16c8cd7 main
   [junit4]   2> 663454 T2123 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 663454 T2123 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 663457 T2152 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@16c8cd7 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 663459 T2123 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
   [junit4]   2> 663459 T2123 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:38432 collection:nodes_used_collection shard:shard1
   [junit4]   2> 663460 T2123 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 663473 T2079 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 663477 T2079 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 663477 T2079 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 663478 T2079 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:53977/nodes_used_collection_shard1_replica1/
   [junit4]   2> 663478 T2079 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:53977 START replicas=[http://127.0.0.1:38432/nodes_used_collection_shard1_replica2/] nUpdates=100
   [junit4]   2> 663479 T2079 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:53977 DONE.  We have no versions.  sync failed.
   [junit4]   2> 663480 T2079 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 663480 T2079 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 663481 T2079 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:53977/nodes_used_collection_shard1_replica1/ shard1
   [junit4]   2> 663481 T2079 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard1
   [junit4]   2> ASYNC  NEW_CORE C19264 name=nodes_used_collection_shard1_replica2 org.apache.solr.core.SolrCore@16c97b9 url=http://127.0.0.1:38432/nodes_used_collection_shard1_replica2 node=127.0.0.1:38432_ C19264_STATE=coll:nodes_used_collection core:nodes_used_collection_shard1_replica2 props:{state=down, core=nodes_used_collection_shard1_replica2, node_name=127.0.0.1:38432_, base_url=http://127.0.0.1:38432}
   [junit4]   2> 663487 T2125 C19264 P38432 oasc.SolrCore.execute [nodes_used_collection_shard1_replica2] webapp= path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
   [junit4]   2> 663900 T2139 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 663900 T2139 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 663901 T2139 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55484/nodes_used_collection_shard2_replica2/
   [junit4]   2> 663902 T2139 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:55484 START replicas=[http://127.0.0.1:36214/nodes_used_collection_shard2_replica1/] nUpdates=100
   [junit4]   2> 663903 T2139 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:55484 DONE.  

[...truncated too long message...]

letesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=1,cumulative_errors=0}
   [junit4]   2> 796897 T2061 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 796897 T2061 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 796898 T2061 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 796899 T2061 oasc.SolrCore.closeSearcher [new_implicit_coll_0_d_replica2] Closing main searcher on request.
   [junit4]   2> 796900 T2061 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 796900 T2061 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//new_implicit_coll_0_d_replica2/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//new_implicit_coll_0_d_replica2/data/index;done=false>>]
   [junit4]   2> 796900 T2061 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//new_implicit_coll_0_d_replica2/data/index
   [junit4]   2> 796901 T2061 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//new_implicit_coll_0_d_replica2/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//new_implicit_coll_0_d_replica2/data;done=false>>]
   [junit4]   2> 796901 T2061 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375479201206//new_implicit_coll_0_d_replica2/data
   [junit4]   2> 796911 T2504 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90143404657803276-127.0.0.1:55484_-n_0000000004) am no longer a leader.
   [junit4]   2> 796941 T2061 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 796995 T2061 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 797000 T2061 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:53607 53607
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=878F5A354C3164F2 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en_PH -Dtests.timezone=Africa/Brazzaville -Dtests.file.encoding=ISO-8859-1
   [junit4] ERROR    144s J1 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:57661
   [junit4]    > at __randomizedtesting.SeedInfo.seed([878F5A354C3164F2:669D42D3B6E04CE]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:764)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > at java.lang.Thread.run(Thread.java:662)
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:57661 refused
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
   [junit4]    > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    > ... 44 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    > at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
   [junit4]    > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
   [junit4]    > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
   [junit4]    > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
   [junit4]    > at java.net.Socket.connect(Socket.java:529)
   [junit4]    > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    > ... 51 more
   [junit4]   2> 797024 T2061 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 143552 T2060 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 798212 T2143 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 798213 T2143 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 798213 T2143 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Asserting, sim=DefaultSimilarity, locale=en_PH, timezone=Africa/Brazzaville
   [junit4]   2> NOTE: Linux 3.2.0-51-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=4,free=62535240,total=194736128
   [junit4]   2> NOTE: All tests run in this JVM: [HighlighterTest, TestQuerySenderNoQuery, SolrIndexConfigTest, TestCSVLoader, TestLMJelinekMercerSimilarityFactory, TestMergePolicyConfig, TestAnalyzedSuggestions, TestIndexSearcher, TestManagedSchemaFieldResource, TestFastOutputStream, TestHashPartitioner, ConvertedLegacyTest, TestSolrJ, HardAutoCommitTest, TestCodecSupport, TestPostingsSolrHighlighter, DateMathParserTest, SimplePostToolTest, TestUtils, TestCollationField, IndexBasedSpellCheckerTest, TestDistributedSearch, URLClassifyProcessorTest, SolrRequestParserTest, OpenExchangeRatesOrgProviderTest, SpellCheckComponentTest, TestZkChroot, PingRequestHandlerTest, DocValuesMultiTest, SampleTest, OpenCloseCoreStressTest, BlockCacheTest, TestDistributedGrouping, HdfsSyncSliceTest, TestCoreDiscovery, DirectUpdateHandlerTest, OutputWriterTest, IndexSchemaRuntimeFieldTest, HdfsCollectionsAPIDistributedZkTest, ZkNodePropsTest, TestElisionMultitermQuery, FileUtilsTest, TestAddFieldRealTimeGet, TestCSVResponseWriter, TestMultiCoreConfBootstrap, TestRandomFaceting, SpellPossibilityIteratorTest, TestJmxMonitoredMap, SignatureUpdateProcessorFactoryTest, TestUpdate, DebugComponentTest, TestFiltering, XmlUpdateRequestHandlerTest, TestReloadAndDeleteDocs, PreAnalyzedUpdateProcessorTest, TestBinaryField, TestBinaryResponseWriter, TestAtomicUpdateErrorCases, CoreAdminCreateDiscoverTest, TestSchemaVersionResource, DistanceFunctionTest, BasicZkTest, ChaosMonkeySafeLeaderTest, DocumentAnalysisRequestHandlerTest, DOMUtilTest, SortByFunctionTest, CurrencyFieldOpenExchangeTest, TermVectorComponentDistributedTest, StandardRequestHandlerTest, SuggesterWFSTTest, RequestHandlersTest, DefaultValueUpdateProcessorTest, TestComponentsName, TestCollationKeyRangeQueries, BasicDistributedZk2Test, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 144.84s, 1 test, 1 error <<< FAILURES!

[...truncated 575 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:375: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: There were test failures: 321 suites, 1352 tests, 1 error, 34 ignored (6 assumptions)

Total time: 45 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -server -XX:+UseConcMarkSweepGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.6.0_45) - Build # 6773 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6773/
Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
REGRESSION:  org.apache.solr.handler.TestReplicationHandler.doTestStressReplication

Error Message:
timed out waiting for collection1 startAt time to exceed: Fri Aug 02 20:15:21 ART 2013

Stack Trace:
java.lang.AssertionError: timed out waiting for collection1 startAt time to exceed: Fri Aug 02 20:15:21 ART 2013
        at __randomizedtesting.SeedInfo.seed([46E57D34E8E5471C:9D4E7DF2EDCD2EAF]:0)
        at org.junit.Assert.fail(Assert.java:93)
        at org.apache.solr.handler.TestReplicationHandler.watchCoreStartAt(TestReplicationHandler.java:1507)
        at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:811)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:662)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Server refused connection at: http://127.0.0.1:42482/cy/wa

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:42482/cy/wa
        at __randomizedtesting.SeedInfo.seed([46E57D34E8E5471C:C703F32C9FBA2720]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:764)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:42482 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        ... 44 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 51 more




Build Log:
[...truncated 9231 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 694541 T2528 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /cy/wa
   [junit4]   2> 694545 T2528 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1375485015066
   [junit4]   2> 694545 T2528 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 694546 T2529 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 694646 T2528 oasc.ZkTestServer.run start zk server on port:55629
   [junit4]   2> 694647 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694697 T2535 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@23cd9ff1 name:ZooKeeperConnection Watcher:127.0.0.1:55629 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694697 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694698 T2528 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 694703 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694704 T2537 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f858887 name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694704 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694704 T2528 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 694708 T2528 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 694712 T2528 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 694715 T2528 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 694718 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 694719 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 694724 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 694724 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 694728 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 694729 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 694732 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 694732 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 694738 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 694738 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 694741 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 694742 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 694745 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 694746 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 694749 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 694749 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 694753 T2528 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 694753 T2528 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 694827 T2528 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 694830 T2528 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43300
   [junit4]   2> 694831 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 694831 T2528 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 694831 T2528 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279
   [junit4]   2> 694832 T2528 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/'
   [junit4]   2> 694842 T2528 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/solr.xml
   [junit4]   2> 694886 T2528 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 694886 T2528 oasc.CoreContainer.<init> New CoreContainer 613319670
   [junit4]   2> 694886 T2528 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/]
   [junit4]   2> 694887 T2528 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 694888 T2528 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 694888 T2528 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 694889 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 694889 T2528 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 694889 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 694890 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 694890 T2528 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 694890 T2528 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 694890 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 694893 T2528 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55629/solr
   [junit4]   2> 694894 T2528 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 694894 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694897 T2548 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8799e3a name:ZooKeeperConnection Watcher:127.0.0.1:55629 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694897 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694899 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 694902 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694903 T2550 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6df610a7 name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694903 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694905 T2528 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 694910 T2528 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 694914 T2528 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 694916 T2528 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43300_cy%2Fwa
   [junit4]   2> 694917 T2528 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43300_cy%2Fwa
   [junit4]   2> 694923 T2528 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 694929 T2528 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 694932 T2528 oasc.Overseer.start Overseer (id=90143785947561987-127.0.0.1:43300_cy%2Fwa-n_0000000000) starting
   [junit4]   2> 694936 T2528 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 694941 T2552 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 694942 T2528 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 694945 T2528 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 694948 T2528 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 694952 T2551 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 694954 T2528 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279
   [junit4]   2> 694987 T2528 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/conf/
   [junit4]   2> 694988 T2528 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 694988 T2528 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 694988 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 694989 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 694991 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 694992 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 694993 T2555 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16574516 name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 694994 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 694995 T2528 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 694998 T2528 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 695062 T2528 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 695063 T2528 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42482
   [junit4]   2> 695064 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 695064 T2528 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 695065 T2528 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375485015520
   [junit4]   2> 695065 T2528 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375485015520/'
   [junit4]   2> 695081 T2528 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375485015520/solr.xml
   [junit4]   2> 695113 T2528 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 695114 T2528 oasc.CoreContainer.<init> New CoreContainer 669756915
   [junit4]   2> 695114 T2528 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375485015520/]
   [junit4]   2> 695115 T2528 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 695115 T2528 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 695115 T2528 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 695116 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 695116 T2528 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 695116 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 695117 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 695117 T2528 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 695117 T2528 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 695118 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 695120 T2528 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55629/solr
   [junit4]   2> 695121 T2528 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 695121 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695123 T2566 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@61e940fe name:ZooKeeperConnection Watcher:127.0.0.1:55629 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695123 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695126 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 695128 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 695129 T2568 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@579169f6 name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 695129 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 695135 T2528 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 696139 T2528 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42482_cy%2Fwa
   [junit4]   2> 696141 T2528 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42482_cy%2Fwa
   [junit4]   2> 696146 T2550 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 696147 T2555 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 696147 T2568 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 696155 T2528 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375485015520
   [junit4]   2> 696179 T2528 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375485015520/conf/
   [junit4]   2> 696179 T2528 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 696180 T2528 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 696180 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 696181 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 696251 T2528 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 696253 T2528 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58557
   [junit4]   2> 696254 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 696254 T2528 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 696254 T2528 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704
   [junit4]   2> 696255 T2528 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/'
   [junit4]   2> 696265 T2528 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/solr.xml
   [junit4]   2> 696296 T2528 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 696296 T2528 oasc.CoreContainer.<init> New CoreContainer 434698780
   [junit4]   2> 696297 T2528 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/]
   [junit4]   2> 696298 T2528 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 696298 T2528 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 696298 T2528 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 696299 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 696299 T2528 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 696299 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 696300 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 696300 T2528 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 696300 T2528 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 696301 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 696303 T2528 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55629/solr
   [junit4]   2> 696304 T2528 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 696305 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696306 T2580 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@e7c518e name:ZooKeeperConnection Watcher:127.0.0.1:55629 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696307 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696309 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 696313 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 696314 T2582 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@16b28576 name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 696314 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 696320 T2528 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 697325 T2528 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:58557_cy%2Fwa
   [junit4]   2> 697328 T2528 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:58557_cy%2Fwa
   [junit4]   2> 697336 T2550 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 697336 T2568 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 697338 T2555 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 697338 T2582 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 697348 T2528 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704
   [junit4]   2> 697378 T2528 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/conf/
   [junit4]   2> 697379 T2528 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 697379 T2528 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 697379 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 697380 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 697457 T2528 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 697459 T2528 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44392
   [junit4]   2> 697459 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 697459 T2528 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 697460 T2528 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903
   [junit4]   2> 697460 T2528 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/'
   [junit4]   2> 697470 T2528 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/solr.xml
   [junit4]   2> 697500 T2528 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 697501 T2528 oasc.CoreContainer.<init> New CoreContainer 1376546892
   [junit4]   2> 697501 T2528 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/]
   [junit4]   2> 697502 T2528 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 697502 T2528 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 697503 T2528 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 697503 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 697503 T2528 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 697503 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 697504 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 697504 T2528 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 697504 T2528 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 697505 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 697507 T2528 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55629/solr
   [junit4]   2> 697508 T2528 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 697508 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 697510 T2594 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@2ea0b7a9 name:ZooKeeperConnection Watcher:127.0.0.1:55629 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 697510 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 697512 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 697515 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 697516 T2596 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@467cb77c name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 697516 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 697521 T2528 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 698526 T2528 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44392_cy%2Fwa
   [junit4]   2> 698528 T2528 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44392_cy%2Fwa
   [junit4]   2> 698534 T2555 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 698534 T2568 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 698534 T2596 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 698536 T2582 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 698536 T2550 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 698549 T2528 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903
   [junit4]   2> 698573 T2528 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/conf/
   [junit4]   2> 698573 T2528 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 698574 T2528 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 698574 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 698575 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 698637 T2528 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 698639 T2528 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:57087
   [junit4]   2> 698639 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 698640 T2528 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 698640 T2528 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098
   [junit4]   2> 698640 T2528 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/'
   [junit4]   2> 698650 T2528 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/solr.xml
   [junit4]   2> 698681 T2528 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 698682 T2528 oasc.CoreContainer.<init> New CoreContainer 569623218
   [junit4]   2> 698682 T2528 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/]
   [junit4]   2> 698683 T2528 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 698683 T2528 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 698684 T2528 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 698684 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 698684 T2528 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 698685 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 698685 T2528 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 698685 T2528 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 698686 T2528 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 698686 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 698695 T2528 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:55629/solr
   [junit4]   2> 698695 T2528 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 698696 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 698697 T2608 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@15d9e914 name:ZooKeeperConnection Watcher:127.0.0.1:55629 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 698698 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 698700 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 698702 T2528 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 698703 T2610 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6d04a538 name:ZooKeeperConnection Watcher:127.0.0.1:55629/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 698704 T2528 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 698710 T2528 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 699716 T2528 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:57087_cy%2Fwa
   [junit4]   2> 699718 T2528 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:57087_cy%2Fwa
   [junit4]   2> 699723 T2582 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 699724 T2596 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 699724 T2550 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 699724 T2610 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 699725 T2555 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 699726 T2568 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 699731 T2528 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098
   [junit4]   2> 699754 T2528 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/conf/
   [junit4]   2> 699755 T2528 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 699756 T2528 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 699756 T2528 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 699756 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 699759 T2528 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 699762 T2564 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 699764 T2550 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 699767 T2552 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 699767 T2552 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 700967 T2551 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 700968 T2551 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: nodes_used_collection
   [junit4]   2> 700968 T2551 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with shards [shard1, shard2]
   [junit4]   2> 700973 T2610 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 700973 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 700973 T2596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 700974 T2550 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 700974 T2568 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 700974 T2582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 701072 T2552 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 701073 T2552 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:58557_cy%2Fwa
   [junit4]   2> 701073 T2552 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:57087_cy%2Fwa
   [junit4]   2> 701074 T2552 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:44392_cy%2Fwa
   [junit4]   2> 701077 T2552 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:43300_cy%2Fwa
   [junit4]   2> 701079 T2577 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
   [junit4]   2> 701079 T2605 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
   [junit4]   2> 701079 T2592 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
   [junit4]   2> 701079 T2544 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
   [junit4]   2> 701081 T2605 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 701081 T2577 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 701082 T2592 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 701082 T2544 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 702485 T2551 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 702487 T2551 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 702487 T2551 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:58557_cy%2Fwa",
   [junit4]   2>  "base_url":"http://127.0.0.1:58557/cy/wa"}
   [junit4]   2> 702494 T2551 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 702494 T2551 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node2",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:57087_cy%2Fwa",
   [junit4]   2>  "base_url":"http://127.0.0.1:57087/cy/wa"}
   [junit4]   2> 702500 T2551 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 702500 T2551 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node3",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:44392_cy%2Fwa",
   [junit4]   2>  "base_url":"http://127.0.0.1:44392/cy/wa"}
   [junit4]   2> 702506 T2551 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 702506 T2551 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node4",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:43300_cy%2Fwa",
   [junit4]   2>  "base_url":"http://127.0.0.1:43300/cy/wa"}
   [junit4]   2> 702511 T2610 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 702512 T2568 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 702512 T2555 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 702511 T2596 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 702511 T2550 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 702512 T2582 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 703081 T2605 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica2
   [junit4]   2> 703082 T2592 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica1
   [junit4]   2> 703082 T2577 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica1
   [junit4]   2> 703083 T2592 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/nodes_used_collection_shard2_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/nodes_used_collection_shard2_replica1/core.properties (No such file or directory)
   [junit4]   2> 703083 T2544 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica2
   [junit4]   2> 703082 T2605 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/nodes_used_collection_shard1_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/nodes_used_collection_shard1_replica2/core.properties (No such file or directory)
   [junit4]   2> 703084 T2544 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/nodes_used_collection_shard2_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/nodes_used_collection_shard2_replica2/core.properties (No such file or directory)
   [junit4]   2> 703083 T2592 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/nodes_used_collection_shard2_replica1
   [junit4]   2> 703084 T2592 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 703083 T2577 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/nodes_used_collection_shard1_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/nodes_used_collection_shard1_replica1/core.properties (No such file or directory)
   [junit4]   2> 703084 T2544 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/nodes_used_collection_shard2_replica2
   [junit4]   2> 703085 T2544 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 703084 T2605 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/nodes_used_collection_shard1_replica2
   [junit4]   2> 703086 T2592 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 703085 T2577 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/nodes_used_collection_shard1_replica1
   [junit4]   2> 703086 T2544 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 703086 T2592 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 703086 T2605 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 703087 T2544 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 703087 T2577 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 703089 T2605 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 703089 T2605 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 703090 T2577 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 703091 T2577 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 703091 T2544 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 703091 T2544 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 703091 T2592 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 703092 T2592 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 703092 T2605 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 703093 T2605 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 703096 T2577 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 703096 T2577 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 703097 T2605 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 703098 T2592 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 703100 T2544 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 703101 T2605 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/nodes_used_collection_shard1_replica2/'
   [junit4]   2> 703101 T2592 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/nodes_used_collection_shard2_replica1/'
   [junit4]   2> 703103 T2577 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 703103 T2544 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/nodes_used_collection_shard2_replica2/'
   [junit4]   2> 703105 T2577 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/nodes_used_collection_shard1_replica1/'
   [junit4]   2> 703160 T2544 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 703174 T2577 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 703177 T2605 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 703182 T2592 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 703242 T2544 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 703245 T2544 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 703246 T2605 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 703249 T2605 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 703272 T2577 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 703272 T2592 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 703275 T2577 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 703277 T2605 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
   [junit4]   2> 703276 T2592 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 703281 T2544 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
   [junit4]   2> 703291 T2577 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
   [junit4]   2> 703294 T2592 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
   [junit4]   2> 703821 T2605 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703821 T2605 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703822 T2605 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 703830 T2605 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 703834 T2605 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 703866 T2605 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 703872 T2605 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 703883 T2605 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 703886 T2605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 703887 T2605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 703887 T2605 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 703889 T2605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 703890 T2605 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 703890 T2605 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 703891 T2605 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098//nodes_used_collection_shard1_replica2/data/
   [junit4]   2> 703891 T2605 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ec78e02
   [junit4]   2> 703893 T2605 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098//nodes_used_collection_shard1_replica2/data
   [junit4]   2> 703893 T2605 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098//nodes_used_collection_shard1_replica2/data/index/
   [junit4]   2> 703894 T2605 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 703894 T2592 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703894 T2605 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375485019098//nodes_used_collection_shard1_replica2/data/index
   [junit4]   2> 703894 T2592 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703895 T2592 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 703897 T2605 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5809c898 lockFactory=org.apache.lucene.store.NativeFSLockFactory@5a3ed429),segFN=segments_1,generation=1}
   [junit4]   2> 703897 T2605 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 703901 T2592 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 703902 T2605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 703903 T2605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 703904 T2605 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 703904 T2592 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 703904 T2605 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 703905 T2605 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 703911 T2605 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 703912 T2605 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 703913 T2605 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 703913 T2605 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 703916 T2605 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 703919 T2605 oass.SolrIndexSearcher.<init> Opening Searcher@379ba3b3 main
   [junit4]   2> 703921 T2605 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 703921 T2605 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 703926 T2616 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@379ba3b3 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 703928 T2605 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
   [junit4]   2> 703929 T2605 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:57087/cy/wa collection:nodes_used_collection shard:shard1
   [junit4]   2> 703929 T2605 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 703933 T2605 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 703936 T2592 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 703943 T2592 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 703945 T2544 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703946 T2544 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703947 T2544 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 703957 T2605 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 703958 T2592 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 703961 T2592 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 703962 T2592 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 703962 T2544 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 703962 T2592 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 703964 T2605 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=179999
   [junit4]   2> 703965 T2544 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 703965 T2592 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 703966 T2592 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 703967 T2592 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 703967 T2592 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903//nodes_used_collection_shard2_replica1/data/
   [junit4]   2> 703967 T2592 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ec78e02
   [junit4]   2> 703969 T2592 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903//nodes_used_collection_shard2_replica1/data
   [junit4]   2> 703970 T2592 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903//nodes_used_collection_shard2_replica1/data/index/
   [junit4]   2> 703970 T2592 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 703971 T2592 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375485017903//nodes_used_collection_shard2_replica1/data/index
   [junit4]   2> 703972 T2577 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703973 T2577 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 703974 T2577 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 703974 T2592 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@69217a9c lockFactory=org.apache.lucene.store.NativeFSLockFactory@10b5b606),segFN=segments_1,generation=1}
   [junit4]   2> 703974 T2592 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 703979 T2592 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 703980 T2592 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 703981 T2592 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 703982 T2592 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 703983 T2577 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 703983 T2592 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 703984 T2592 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 703984 T2592 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 703985 T2592 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 703986 T2592 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 703987 T2577 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 703987 T2544 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 703989 T2592 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 703991 T2592 oass.SolrIndexSearcher.<init> Opening Searcher@a28ad2e main
   [junit4]   2> 703992 T2592 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 703993 T2592 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 703993 T2544 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 703996 T2617 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@a28ad2e main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 703997 T2592 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
   [junit4]   2> 703998 T2592 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:44392/cy/wa collection:nodes_used_collection shard:shard2
   [junit4]   2> 703998 T2544 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 703998 T2592 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 704001 T2544 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 704001 T2544 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 704001 T2544 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 704003 T2592 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
   [junit4]   2> 704006 T2544 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 704006 T2544 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 704007 T2544 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 704007 T2544 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279//nodes_used_collection_shard2_replica2/data/
   [junit4]   2> 704007 T2544 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ec78e02
   [junit4]   2> 704008 T2544 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279//nodes_used_collection_shard2_replica2/data
   [junit4]   2> 704009 T2544 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279//nodes_used_collection_shard2_replica2/data/index/
   [junit4]   2> 704009 T2544 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 704009 T2544 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375485015279//nodes_used_collection_shard2_replica2/data/index
   [junit4]   2> 704011 T2544 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@5008387c lockFactory=org.apache.lucene.store.NativeFSLockFactory@7a08334c),segFN=segments_1,generation=1}
   [junit4]   2> 704011 T2544 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 704014 T2577 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 704015 T2544 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 704015 T2544 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 704016 T2544 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 704016 T2544 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 704017 T2544 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 704017 T2544 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 704018 T2544 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 704018 T2544 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 704019 T2544 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 704019 T2592 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 704020 T2577 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 704020 T2544 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 704023 T2544 oass.SolrIndexSearcher.<init> Opening Searcher@190f51a main
   [junit4]   2> 704024 T2544 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 704024 T2544 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 704025 T2592 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=179999
   [junit4]   2> 704026 T2577 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 704027 T2618 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@190f51a main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 704028 T2577 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 704028 T2577 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 704028 T2544 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
   [junit4]   2> 704029 T2577 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 704029 T2544 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:43300/cy/wa collection:nodes_used_collection shard:shard2
   [junit4]   2> 704030 T2544 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 704031 T2551 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 704031 T2577 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 704032 T2577 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 704032 T2577 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 704033 T2577 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704//nodes_used_collection_shard1_replica1/data/
   [junit4]   2> 704033 T2577 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@7ec78e02
   [junit4]   2> 704034 T2577 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704//nodes_used_collection_shard1_replica1/data
   [junit4]   2> 704043 T2577 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704//nodes_used_collection_shard1_replica1/data/index/
   [junit4]   2> 704043 T2577 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 704044 T2577 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375485016704//nodes_used_collection_shard1_replica1/data/index
   [junit4]   2> 704046 T2577 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@2882f357 lockFactory=org.apache.lucene.store.NativeFSLockFactory@44c112d),segFN=segments_1,generation=1}
   [junit4]   2> 704046 T2577 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 704050 T2577 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 704050 T257

[...truncated too long message...]

junit4]   2> 1044565 T3230 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364542/repeater/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364542/repeater/collection1/data;done=false>>]
   [junit4]   2> 1044566 T3230 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364542/repeater/collection1/data
   [junit4]   2> 1044566 T3230 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1044618 T3230 oas.SolrTestCaseJ4.tearDown ###Ending doTestRepeater
   [junit4]   2> 1044639 T3230 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=170967737
   [junit4]   2> 1044639 T3230 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@253428ab
   [junit4]   2> 1044640 T3230 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=2,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=6,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1044641 T3230 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1044641 T3230 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1044641 T3230 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1044642 T3230 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1044643 T3230 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1044643 T3230 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364022/master/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364022/master/collection1/data;done=false>>]
   [junit4]   2> 1044643 T3230 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364022/master/collection1/data
   [junit4]   2> 1044644 T3230 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364022/master/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364022/master/collection1/data/index;done=false>>]
   [junit4]   2> 1044644 T3230 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364022/master/collection1/data/index
   [junit4]   2> 1044644 T3230 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1044717 T3230 oasc.CoreContainer.shutdown Shutting down CoreContainer instance=319961884
   [junit4]   2> 1044717 T3230 oasc.SolrCore.close [collection1]  CLOSING SolrCore org.apache.solr.core.SolrCore@9431b98
   [junit4]   2> 1044718 T3230 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 1044718 T3230 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 1044719 T3230 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 1044719 T3230 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 1044720 T3230 oasc.SolrCore.closeSearcher [collection1] Closing main searcher on request.
   [junit4]   2> 1044720 T3230 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 1044721 T3230 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364182/slave/collection1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364182/slave/collection1/data/index;done=false>>]
   [junit4]   2> 1044721 T3230 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364182/slave/collection1/data/index
   [junit4]   2> 1044721 T3230 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364182/slave/collection1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364182/slave/collection1/data;done=false>>]
   [junit4]   2> 1044722 T3230 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.handler.TestReplicationHandler$SolrInstance-1375485364182/slave/collection1/data
   [junit4]   2> 1044722 T3230 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/solr,null}
   [junit4]   2> 1044777 T3230 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=tr_TR, timezone=America/Argentina/Tucuman
   [junit4]   2> NOTE: Linux 3.2.0-51-generic amd64/Sun Microsystems Inc. 1.6.0_45 (64-bit)/cpus=8,threads=1,free=85895968,total=192897024
   [junit4]   2> NOTE: All tests run in this JVM: [PreAnalyzedFieldTest, DocValuesTest, HdfsUnloadDistributedZkTest, BasicFunctionalityTest, CachingDirectoryFactoryTest, TestSchemaNameResource, SOLR749Test, BadComponentTest, MBeansHandlerTest, SolrInfoMBeanTest, TestSolrXmlPersistence, BinaryUpdateRequestHandlerTest, SimpleFacetsTest, TestRandomDVFaceting, TestSolrQueryParser, UpdateRequestProcessorFactoryTest, TestQueryUtils, QueryParsingTest, TestSearchPerf, SynonymTokenizerTest, IndexSchemaTest, TestRealTimeGet, TermVectorComponentTest, PrimitiveFieldTypeTest, SoftAutoCommitTest, DistributedTermsComponentTest, ZkControllerTest, UnloadDistributedZkTest, TestPseudoReturnFields, PolyFieldTest, OverseerTest, TestDefaultSearchFieldResource, CircularListTest, DistributedQueryElevationComponentTest, TestRemoteStreaming, TestDynamicFieldCollectionResource, SyncSliceTest, AlternateDirectoryTest, TestSolrQueryParserResource, TestFieldTypeResource, TestLMDirichletSimilarityFactory, TestSolrXml, DistributedSpellCheckComponentTest, TestDocumentBuilder, FastVectorHighlighterTest, TestBadConfig, PeerSyncTest, TestGroupingSearch, ScriptEngineTest, TestReversedWildcardFilterFactory, TestSweetSpotSimilarityFactory, CSVRequestHandlerTest, ResponseLogComponentTest, TestFastLRUCache, LeaderElectionTest, ShowFileRequestHandlerTest, TestIBSimilarityFactory, ShardRoutingTest, RegexBoostProcessorTest, DirectSolrConnectionTest, ZkCLITest, FieldMutatingUpdateProcessorTest, UUIDFieldTest, StressHdfsTest, ReturnFieldsTest, TestStressVersions, RecoveryZkTest, TestPhraseSuggestions, TestFuzzyAnalyzedSuggestions, DirectUpdateHandlerOptimizeTest, TestPropInjectDefaults, HdfsChaosMonkeySafeLeaderTest, DateFieldTest, TestCoreContainer, TestExtendedDismaxParser, SuggesterTest, AliasIntegrationTest, BasicDistributedZkTest, TestReplicationHandler]
   [junit4] Completed on J0 in 63.68s, 14 tests, 1 failure <<< FAILURES!

[...truncated 472 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:375: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: There were test failures: 321 suites, 1352 tests, 1 error, 1 failure, 34 ignored (6 assumptions)

Total time: 41 minutes 16 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.6.0_45 -XX:-UseCompressedOops -XX:+UseSerialGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-4.x-Linux (64bit/jdk1.7.0_25) - Build # 6774 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6774/
Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Server refused connection at: http://127.0.0.1:54133/c/km

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:54133/c/km
        at __randomizedtesting.SeedInfo.seed([F83977DF27507C93:79DFF9C7500F1CAF]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1535)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:677)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:724)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:54133 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        ... 45 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 52 more




Build Log:
[...truncated 9684 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 228117 T1136 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /c/km
   [junit4]   2> 228123 T1136 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-CollectionsAPIDistributedZkTest-1375489800139
   [junit4]   2> 228124 T1136 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 228125 T1137 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 228225 T1136 oasc.ZkTestServer.run start zk server on port:47719
   [junit4]   2> 228226 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228265 T1143 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@596fded5 name:ZooKeeperConnection Watcher:127.0.0.1:47719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228266 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228266 T1136 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 228277 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228278 T1145 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@57b8fec4 name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228279 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228279 T1136 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 228286 T1136 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 228289 T1136 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 228291 T1136 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 228296 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 228297 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 228304 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 228305 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 228309 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 228310 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 228319 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 228320 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 228325 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 228326 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 228328 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 228329 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 228336 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 228336 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 228341 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 228341 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 228353 T1136 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 228353 T1136 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 228443 T1136 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 228445 T1136 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:35468
   [junit4]   2> 228448 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 228448 T1136 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 228449 T1136 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374
   [junit4]   2> 228449 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/'
   [junit4]   2> 228460 T1136 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/solr.xml
   [junit4]   2> 228496 T1136 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 228496 T1136 oasc.CoreContainer.<init> New CoreContainer 46112790
   [junit4]   2> 228497 T1136 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/]
   [junit4]   2> 228498 T1136 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 228498 T1136 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 228498 T1136 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 228498 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 228499 T1136 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 228499 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 228499 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 228499 T1136 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 228500 T1136 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 228500 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 228503 T1136 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47719/solr
   [junit4]   2> 228504 T1136 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 228505 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228506 T1156 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@45cb2463 name:ZooKeeperConnection Watcher:127.0.0.1:47719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228506 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228508 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 228512 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228514 T1158 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@4da484d4 name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228514 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228516 T1136 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 228519 T1136 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 228522 T1136 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 228524 T1136 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:35468_c%2Fkm
   [junit4]   2> 228525 T1136 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:35468_c%2Fkm
   [junit4]   2> 228528 T1136 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 228533 T1136 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 228536 T1136 oasc.Overseer.start Overseer (id=90144099542106115-127.0.0.1:35468_c%2Fkm-n_0000000000) starting
   [junit4]   2> 228538 T1136 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 228541 T1160 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 228542 T1136 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 228544 T1136 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 228546 T1136 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 228548 T1159 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 228551 T1136 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374
   [junit4]   2> 228580 T1136 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/conf/
   [junit4]   2> 228580 T1136 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 228581 T1136 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 228581 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 228582 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 228586 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 228588 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228589 T1163 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20738ace name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228589 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228590 T1136 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 228593 T1136 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 228696 T1136 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 228699 T1136 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54133
   [junit4]   2> 228700 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 228700 T1136 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 228701 T1136 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608
   [junit4]   2> 228701 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/'
   [junit4]   2> 228718 T1136 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/solr.xml
   [junit4]   2> 228768 T1136 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 228768 T1136 oasc.CoreContainer.<init> New CoreContainer 579377132
   [junit4]   2> 228769 T1136 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/]
   [junit4]   2> 228770 T1136 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 228771 T1136 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 228771 T1136 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 228771 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 228772 T1136 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 228772 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 228772 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 228773 T1136 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 228773 T1136 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 228774 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 228779 T1136 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47719/solr
   [junit4]   2> 228780 T1136 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 228781 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228785 T1174 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@3fa756e8 name:ZooKeeperConnection Watcher:127.0.0.1:47719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228786 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228788 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 228792 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 228799 T1176 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@25268e11 name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 228799 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 228804 T1136 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 229806 T1136 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54133_c%2Fkm
   [junit4]   2> 229808 T1136 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54133_c%2Fkm
   [junit4]   2> 229810 T1158 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 229810 T1176 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 229810 T1163 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 229815 T1136 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608
   [junit4]   2> 229843 T1136 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/conf/
   [junit4]   2> 229843 T1136 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 229844 T1136 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 229844 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 229844 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 229929 T1136 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 229931 T1136 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:55204
   [junit4]   2> 229932 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 229932 T1136 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 229932 T1136 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863
   [junit4]   2> 229933 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/'
   [junit4]   2> 229944 T1136 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/solr.xml
   [junit4]   2> 229981 T1136 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 229981 T1136 oasc.CoreContainer.<init> New CoreContainer 755681361
   [junit4]   2> 229981 T1136 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/]
   [junit4]   2> 229983 T1136 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 229983 T1136 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 229983 T1136 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 229983 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 229984 T1136 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 229984 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 229984 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 229985 T1136 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 229985 T1136 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 229985 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 229988 T1136 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47719/solr
   [junit4]   2> 229989 T1136 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 229990 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 229991 T1188 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@20a9f123 name:ZooKeeperConnection Watcher:127.0.0.1:47719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 229991 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 229994 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 229998 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 229999 T1190 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@7f34e8ee name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 229999 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 230002 T1136 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 231005 T1136 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:55204_c%2Fkm
   [junit4]   2> 231007 T1136 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:55204_c%2Fkm
   [junit4]   2> 231009 T1158 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 231009 T1190 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 231010 T1176 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 231015 T1136 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863
   [junit4]   2> 231016 T1163 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 231042 T1136 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/conf/
   [junit4]   2> 231042 T1136 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 231043 T1136 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 231043 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 231043 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 231135 T1136 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 231146 T1136 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:43440
   [junit4]   2> 231147 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 231148 T1136 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 231148 T1136 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061
   [junit4]   2> 231148 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/'
   [junit4]   2> 231164 T1136 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/solr.xml
   [junit4]   2> 231208 T1136 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 231209 T1136 oasc.CoreContainer.<init> New CoreContainer 235892563
   [junit4]   2> 231209 T1136 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/]
   [junit4]   2> 231210 T1136 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 231211 T1136 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 231211 T1136 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 231211 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 231211 T1136 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 231212 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 231212 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 231212 T1136 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 231213 T1136 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 231213 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 231218 T1136 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47719/solr
   [junit4]   2> 231219 T1136 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 231220 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 231222 T1202 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8d5e1e7 name:ZooKeeperConnection Watcher:127.0.0.1:47719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 231222 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 231224 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 231229 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 231231 T1204 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@701a2d0b name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 231231 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 231236 T1136 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 232242 T1136 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:43440_c%2Fkm
   [junit4]   2> 232243 T1136 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:43440_c%2Fkm
   [junit4]   2> 232245 T1190 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 232246 T1204 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 232246 T1158 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 232246 T1163 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 232246 T1176 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 232251 T1136 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061
   [junit4]   2> 232285 T1136 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/conf/
   [junit4]   2> 232285 T1136 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 232286 T1136 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 232286 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 232287 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 232361 T1136 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 232363 T1136 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:40100
   [junit4]   2> 232364 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 232365 T1136 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 232365 T1136 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307
   [junit4]   2> 232366 T1136 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307/'
   [junit4]   2> 232380 T1136 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307/solr.xml
   [junit4]   2> 232426 T1136 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 232426 T1136 oasc.CoreContainer.<init> New CoreContainer 48047830
   [junit4]   2> 232427 T1136 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307/]
   [junit4]   2> 232428 T1136 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 232429 T1136 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 232429 T1136 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 232430 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 232430 T1136 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 232430 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 232431 T1136 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 232431 T1136 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 232432 T1136 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 232432 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 232436 T1136 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:47719/solr
   [junit4]   2> 232436 T1136 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 232438 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 232439 T1216 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@8b1ac92 name:ZooKeeperConnection Watcher:127.0.0.1:47719 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 232439 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 232442 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 232446 T1136 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 232453 T1218 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@12398f0f name:ZooKeeperConnection Watcher:127.0.0.1:47719/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 232454 T1136 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 232466 T1136 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 233469 T1136 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:40100_c%2Fkm
   [junit4]   2> 233470 T1136 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:40100_c%2Fkm
   [junit4]   2> 233473 T1158 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 233474 T1190 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 233474 T1163 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 233474 T1218 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 233474 T1176 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 233474 T1204 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 233480 T1136 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307
   [junit4]   2> 233519 T1136 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307/conf/
   [junit4]   2> 233520 T1136 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 233520 T1136 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 233521 T1136 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 233521 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 233524 T1136 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 233529 T1169 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 233530 T1158 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 233533 T1160 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 233533 T1160 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 234561 T1159 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 234562 T1159 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: nodes_used_collection
   [junit4]   2> 234562 T1159 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with shards [shard1, shard2]
   [junit4]   2> 234568 T1218 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 234568 T1176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 234568 T1158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 234568 T1190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 234568 T1204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 234568 T1163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 234637 T1160 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 234637 T1160 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:54133_c%2Fkm
   [junit4]   2> 234638 T1160 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:35468_c%2Fkm
   [junit4]   2> 234638 T1160 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:43440_c%2Fkm
   [junit4]   2> 234638 T1160 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:55204_c%2Fkm
   [junit4]   2> 234654 T1154 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
   [junit4]   2> 234654 T1197 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
   [junit4]   2> 234654 T1172 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
   [junit4]   2> 234654 T1183 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
   [junit4]   2> 234660 T1154 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 234660 T1183 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 234660 T1172 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 234660 T1197 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 236077 T1159 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 236078 T1159 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 236078 T1159 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:35468_c%2Fkm",
   [junit4]   2>  "base_url":"http://127.0.0.1:35468/c/km"}
   [junit4]   2> 236082 T1159 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 236082 T1159 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node2",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:43440_c%2Fkm",
   [junit4]   2>  "base_url":"http://127.0.0.1:43440/c/km"}
   [junit4]   2> 236086 T1159 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 236086 T1159 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node3",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:54133_c%2Fkm",
   [junit4]   2>  "base_url":"http://127.0.0.1:54133/c/km"}
   [junit4]   2> 236096 T1159 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 236097 T1159 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node4",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:55204_c%2Fkm",
   [junit4]   2>  "base_url":"http://127.0.0.1:55204/c/km"}
   [junit4]   2> 236100 T1163 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 236101 T1218 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 236101 T1204 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 236102 T1190 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 236102 T1176 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 236103 T1158 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 236660 T1154 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica2
   [junit4]   2> 236661 T1172 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica1
   [junit4]   2> 236661 T1183 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica2
   [junit4]   2> 236665 T1183 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/nodes_used_collection_shard2_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/nodes_used_collection_shard2_replica2/core.properties (No such file or directory)
   [junit4]   2> 236664 T1197 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica1
   [junit4]   2> 236664 T1172 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/nodes_used_collection_shard1_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/nodes_used_collection_shard1_replica1/core.properties (No such file or directory)
   [junit4]   2> 236661 T1154 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/nodes_used_collection_shard1_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/nodes_used_collection_shard1_replica2/core.properties (No such file or directory)
   [junit4]   2> 236666 T1197 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/nodes_used_collection_shard2_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/nodes_used_collection_shard2_replica1/core.properties (No such file or directory)
   [junit4]   2> 236666 T1172 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/nodes_used_collection_shard1_replica1
   [junit4]   2> 236667 T1172 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 236666 T1183 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/nodes_used_collection_shard2_replica2
   [junit4]   2> 236667 T1197 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/nodes_used_collection_shard2_replica1
   [junit4]   2> 236668 T1197 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 236667 T1154 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/nodes_used_collection_shard1_replica2
   [junit4]   2> 236668 T1172 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 236669 T1197 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 236668 T1183 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 236669 T1197 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 236669 T1172 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 236669 T1154 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 236670 T1183 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 236671 T1183 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 236671 T1197 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 236671 T1197 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 236672 T1172 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 236672 T1172 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 236671 T1154 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 236672 T1154 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 236673 T1183 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 236673 T1183 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 236674 T1154 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 236674 T1154 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 236675 T1197 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 236676 T1172 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 236676 T1183 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 236676 T1154 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 236677 T1172 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/nodes_used_collection_shard1_replica1/'
   [junit4]   2> 236676 T1197 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/nodes_used_collection_shard2_replica1/'
   [junit4]   2> 236680 T1183 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/nodes_used_collection_shard2_replica2/'
   [junit4]   2> 236680 T1154 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/nodes_used_collection_shard1_replica2/'
   [junit4]   2> 236771 T1154 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 236813 T1197 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 236827 T1183 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 236833 T1172 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 236855 T1154 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 236857 T1154 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 236869 T1154 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
   [junit4]   2> 236870 T1183 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 236871 T1183 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 236881 T1183 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
   [junit4]   2> 236888 T1172 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 236889 T1197 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 236891 T1172 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 236892 T1197 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 236908 T1172 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
   [junit4]   2> 236909 T1197 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
   [junit4]   2> 237671 T1183 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237672 T1183 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237673 T1183 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 237680 T1183 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 237684 T1183 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 237710 T1183 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237715 T1183 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237719 T1183 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237721 T1183 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237722 T1183 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237728 T1183 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237730 T1183 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237731 T1183 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237732 T1183 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 237732 T1183 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863//nodes_used_collection_shard2_replica2/data/
   [junit4]   2> 237732 T1183 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@446caed6
   [junit4]   2> 237733 T1183 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863//nodes_used_collection_shard2_replica2/data
   [junit4]   2> 237734 T1183 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863//nodes_used_collection_shard2_replica2/data/index/
   [junit4]   2> 237740 T1183 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 237740 T1183 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375489801863//nodes_used_collection_shard2_replica2/data/index
   [junit4]   2> 237743 T1183 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@13d21f46 lockFactory=org.apache.lucene.store.NativeFSLockFactory@242392f7),segFN=segments_1,generation=1}
   [junit4]   2> 237744 T1183 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 237749 T1183 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 237750 T1183 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 237751 T1183 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 237751 T1183 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 237752 T1183 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 237752 T1183 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 237753 T1183 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 237754 T1183 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 237754 T1183 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 237758 T1183 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 237776 T1183 oass.SolrIndexSearcher.<init> Opening Searcher@76046361 main
   [junit4]   2> 237781 T1183 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 237781 T1183 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 237786 T1224 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@76046361 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 237788 T1183 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
   [junit4]   2> 237788 T1183 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:55204/c/km collection:nodes_used_collection shard:shard2
   [junit4]   2> 237788 T1183 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 237795 T1183 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
   [junit4]   2> 237805 T1183 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 237817 T1154 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237817 T1154 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237818 T1154 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 237820 T1183 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=179999
   [junit4]   2> 237826 T1154 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 237830 T1154 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 237831 T1197 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237831 T1197 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237832 T1197 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 237839 T1197 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 237842 T1172 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237842 T1197 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 237842 T1172 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 237844 T1172 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 237849 T1172 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 237851 T1172 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 237855 T1154 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237861 T1154 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237863 T1197 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237865 T1154 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237868 T1154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237868 T1154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237868 T1172 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237868 T1154 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237868 T1197 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237872 T1154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237872 T1154 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237873 T1154 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 237873 T1154 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374//nodes_used_collection_shard1_replica2/data/
   [junit4]   2> 237874 T1154 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@446caed6
   [junit4]   2> 237875 T1197 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237875 T1172 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 237875 T1154 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374//nodes_used_collection_shard1_replica2/data
   [junit4]   2> 237876 T1154 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374//nodes_used_collection_shard1_replica2/data/index/
   [junit4]   2> 237876 T1154 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 237876 T1197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237877 T1197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237877 T1154 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375489800374//nodes_used_collection_shard1_replica2/data/index
   [junit4]   2> 237879 T1172 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237881 T1154 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@8c0cba1 lockFactory=org.apache.lucene.store.NativeFSLockFactory@6af0efa6),segFN=segments_1,generation=1}
   [junit4]   2> 237877 T1197 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237881 T1154 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 237881 T1172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237883 T1172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237883 T1197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237883 T1197 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237883 T1172 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 237884 T1197 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 237885 T1197 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061//nodes_used_collection_shard2_replica1/data/
   [junit4]   2> 237885 T1197 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@446caed6
   [junit4]   2> 237886 T1172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 237886 T1172 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 237887 T1172 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 237886 T1154 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 237887 T1172 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608//nodes_used_collection_shard1_replica1/data/
   [junit4]   2> 237887 T1172 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@446caed6
   [junit4]   2> 237887 T1197 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061//nodes_used_collection_shard2_replica1/data
   [junit4]   2> 237887 T1154 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 237888 T1172 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608//nodes_used_collection_shard1_replica1/data
   [junit4]   2> 237888 T1197 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061//nodes_used_collection_shard2_replica1/data/index/
   [junit4]   2> 237889 T1172 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608//nodes_used_collection_shard1_replica1/data/index/
   [junit4]   2> 237889 T1197 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 237889 T1154 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 237889 T1172 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 237890 T1197 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375489803061//nodes_used_collection_shard2_replica1/data/index
   [junit4]   2> 237890 T1172 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375489800608//nodes_used_collection_shard1_replica1/data/index
   [junit4]   2> 237890 T1154 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 237892 T1154 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 237892 T1154 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 237892 T1197 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@7ee2b289 lockFactory=org.apache.lucene.store.NativeFSLockFactory@3dec7b7b),segFN=segments_1,generation=1}
   [junit4]   2> 237892 T1197 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 237892 T1154 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 237895 T1154 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 237895 T1154 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 237896 T1197 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 237896 T1197 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 237897 T1197 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 237897 T1197 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 237898 T1154 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 237898 T1197 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 237898 T1197 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 237898 T1197 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 237899 T1197 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 237899 T1197 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 237899 T1172 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@74228796 lockFactory=org.apache.lucene.store.NativeFSLockFactory@28584208),segFN=segments_1,generation=1}
   [junit4]   2> 237900 T1197 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 237901 T1172 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 237901 T1154 oass.SolrIndexSearcher.<init> Opening Searcher@5b98a346 main
   [junit4]   2> 237902 T1154 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 237902 T1154 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 237903 T1197 oass.SolrIndexSearcher.<init> Opening Searcher@6b7f37aa main
   [junit4]   2> 237904 T1197 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 237904 T1197 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 237906 T1172 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 237906 T1154 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
   [junit4]   2> 237906 T1154 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:35468/c/km collection:nodes_used_collection shard:shard1
   [junit4]   2> 237906 T1172 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 237907 T1154 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 237907 T1225 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@5b98a346 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 237907 T1172 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 237908 T1226 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@6b7f37aa main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 237909 T1172 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 237909 T1172 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 237909 T1172 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 237909 T1197 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
   [junit4]   2> 237910 T1172 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 237910 T1197 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:43440/c/km collection:nodes_used_collection shard:shard2
   [junit4]   2> 237911 T1197 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 237911 T1172 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 237912 T1154 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 237912 T1172 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 237922 T1154 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 237922 T1172 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 237925 T1154 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=180000
   [junit4]   2> 237926 T1172 oass.SolrIndexSearcher.<init> Opening Searcher@1146ac9a main
   [junit4]   2> 237927 T1172 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 237927 T1172 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 237932 T1172 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
   [junit4]   2> 237932 T1172 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:54133/c/km collection:nodes_used_collection shard:shard1
   [junit4]   2> 237932 T1172 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 237932 T1227 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@1146ac9a main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 238321 T1183 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 238322 T1183 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 238322 T1183 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:55204/c/km/nodes_used_collection_shard2_replica2/
   [junit4]   2> 238323 T1183 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:55204/c/km START replicas=[http://127.0.0.1:43440/c/km/nodes_used_collection_shard2_replica1/] nUpdates=100
   [junit4]   2> 238323 T1183 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard2_replica2 url=http://127.0.0.1:55204/c/km DONE.  We have no versions.  sync failed.
   [junit4]   2> 238324 T1183 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 238324 T1183 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 238324 T1183 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:55204/c/km/nodes_used_collection_shard2_replica2/ shard2
   [junit4]   2> 238325 T1183 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leaders/shard2
   [junit4]   2> ASYNC  NEW_CORE C247 name=nodes_used_collection_shard2_replica1 org.apache.solr.core.SolrCore@142d88b0 url=http://127.0.0.1:43440/c/km/nodes_used_collection_shard2_replica1 node=127.0.0.1:43440_c%2Fkm C247_STATE=coll:nodes_used_collection core:nodes_used_collection_shard2_replica1 props:{state=down, core=nodes_used_collection_shard2_replica1, node_name=127.0.0.1:43440_c%2Fkm, base_url=http://127.0.0.1:43440/c/km}
   [junit4]   2> 238331 T1198 C247 P43440 oasc.SolrCore.execute [nodes_used_collection_shard2_replica1] webapp=/c/km path=/get params={getVersions=100&distrib=false&wt=javabin&qt=/get&version=2} status=0 QTime=1
   [junit4]   2> 238427 T1154 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 238427 T1154 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 238428 T1154 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:35468/c/km/nodes_used_collection_shard1_replica2/
   [junit4]   2> 238428 T11

[...truncated too long message...]

oll_1_d_replica1/data;done=false>>]
   [junit4]   2> 333625 T1136 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_d_replica1/data
   [junit4]   2> 333625 T1136 oasc.SolrCore.close [new_implicit_coll_1_c_replica1]  CLOSING SolrCore org.apache.solr.core.SolrCore@a03b240
   [junit4]   2> 333631 T1136 oasu.DirectUpdateHandler2.close closing DirectUpdateHandler2{commits=0,autocommits=0,soft autocommits=0,optimizes=0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 333631 T1136 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 333632 T1136 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 333632 T1136 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 333632 T1136 oasc.SolrCore.closeSearcher [new_implicit_coll_1_c_replica1] Closing main searcher on request.
   [junit4]   2> 333632 T1136 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 333633 T1136 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_c_replica1/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_c_replica1/data/index;done=false>>]
   [junit4]   2> 333633 T1136 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_c_replica1/data/index
   [junit4]   2> 333633 T1136 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_c_replica1/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_c_replica1/data;done=false>>]
   [junit4]   2> 333633 T1136 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375489804307//new_implicit_coll_1_c_replica1/data
   [junit4]   2> 333634 T1484 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90144099542106124-127.0.0.1:40100_c%2Fkm-n_0000000004) am no longer a leader.
   [junit4]   2> 333657 T1136 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/c/km,null}
   [junit4]   2> 333711 T1136 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 333713 T1136 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:47719 47719
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=F83977DF27507C93 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=es -Dtests.timezone=Etc/GMT0 -Dtests.file.encoding=US-ASCII
   [junit4] ERROR    106s J1 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:54133/c/km
   [junit4]    > at __randomizedtesting.SeedInfo.seed([F83977DF27507C93:79DFF9C7500F1CAF]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1535)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:677)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > at java.lang.Thread.run(Thread.java:724)
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:54133 refused
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
   [junit4]    > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    > ... 45 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    > at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
   [junit4]    > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
   [junit4]    > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
   [junit4]    > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
   [junit4]    > at java.net.Socket.connect(Socket.java:579)
   [junit4]    > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    > ... 52 more
   [junit4]   2> 333732 T1136 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 105620 T1135 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 334936 T1218 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 334936 T1218 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 334936 T1218 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=es, timezone=Etc/GMT0
   [junit4]   2> NOTE: Linux 3.2.0-51-generic amd64/Oracle Corporation 1.7.0_25 (64-bit)/cpus=8,threads=2,free=232050904,total=325058560
   [junit4]   2> NOTE: All tests run in this JVM: [ChaosMonkeySafeLeaderTest, TestArbitraryIndexDir, SortByFunctionTest, IndexSchemaRuntimeFieldTest, TestDistributedSearch, TestSuggestSpellingConverter, CopyFieldTest, IndexBasedSpellCheckerTest, TestIndexSearcher, TestRTGBase, PingRequestHandlerTest, TestCloudManagedSchema, TestSchemaVersionResource, TestPartialUpdateDeduplication, TestElisionMultitermQuery, TestLMJelinekMercerSimilarityFactory, RequestHandlersTest, TestCoreDiscovery, TestBinaryResponseWriter, SuggesterTSTTest, TestAddFieldRealTimeGet, DirectSolrSpellCheckerTest, SolrRequestParserTest, TestSolrJ, TestSchemaSimilarityResource, NumericFieldsTest, TestCollationKeyRangeQueries, HdfsSyncSliceTest, TestCSVLoader, TestStressReorder, TestReloadAndDeleteDocs, DeleteShardTest, TestAnalyzedSuggestions, AssignTest, TestFiltering, TestManagedSchemaFieldResource, ZkNodePropsTest, TestCSVResponseWriter, HighlighterTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 106.90s, 1 test, 1 error <<< FAILURES!

[...truncated 733 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:375: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: There were test failures: 321 suites, 1352 tests, 1 error, 34 ignored (6 assumptions)

Total time: 42 minutes 59 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 64bit/jdk1.7.0_25 -XX:+UseCompressedOops -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-4.x-Linux (32bit/ibm-j9-jdk7) - Build # 6775 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6775/
Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}

2 tests failed.
REGRESSION:  org.apache.solr.cloud.BasicDistributedZk2Test.testDistribSearch

Error Message:
Server at http://127.0.0.1:38311/cif/bc/onenodecollectioncore returned non ok status:404, message:Can not find: /cif/bc/onenodecollectioncore/update

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Server at http://127.0.0.1:38311/cif/bc/onenodecollectioncore returned non ok status:404, message:Can not find: /cif/bc/onenodecollectioncore/update
        at __randomizedtesting.SeedInfo.seed([9790304D47D73F71:1676BE5530885F4D]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:385)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116)
        at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102)
        at org.apache.solr.cloud.BasicDistributedZk2Test.testNodeWithoutCollectionForwarding(BasicDistributedZk2Test.java:196)
        at org.apache.solr.cloud.BasicDistributedZk2Test.doTest(BasicDistributedZk2Test.java:88)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:613)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:780)


FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Server refused connection at: http://127.0.0.1:56594

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:56594
        at __randomizedtesting.SeedInfo.seed([9790304D47D73F71:1676BE5530885F4D]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1535)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:677)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:88)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
        at java.lang.reflect.Method.invoke(Method.java:613)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:780)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:56594 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        ... 45 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:369)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
        at java.net.Socket.connect(Socket.java:590)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 52 more




Build Log:
[...truncated 9618 lines...]
   [junit4] Suite: org.apache.solr.cloud.BasicDistributedZk2Test
   [junit4]   2> 258988 T699 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /cif/bc
   [junit4]   2> 258994 T699 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./solrtest-BasicDistributedZk2Test-1375494996738
   [junit4]   2> 258995 T699 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 258996 T700 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 259096 T699 oasc.ZkTestServer.run start zk server on port:43537
   [junit4]   2> 259100 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 259213 T706 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@34daf02 name:ZooKeeperConnection Watcher:127.0.0.1:43537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 259213 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 259214 T699 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 259221 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 259228 T708 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c54d91c9 name:ZooKeeperConnection Watcher:127.0.0.1:43537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 259229 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 259230 T699 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 259237 T699 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 259242 T699 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 259247 T699 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 259251 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 259252 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 259257 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 259258 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 259262 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 259264 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 259266 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 259267 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 259270 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 259271 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 259274 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 259275 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 259278 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 259279 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 259282 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 259283 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 259286 T699 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 259287 T699 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 259416 T699 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 259426 T699 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:50783
   [junit4]   2> 259427 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 259428 T699 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 259429 T699 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035
   [junit4]   2> 259430 T699 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/'
   [junit4]   2> 259442 T699 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/solr.xml
   [junit4]   2> 259494 T699 oasc.CoreContainer.<init> New CoreContainer 1925697923
   [junit4]   2> 259496 T699 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/]
   [junit4]   2> 259498 T699 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 259499 T699 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 259501 T699 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 259502 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 259503 T699 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 259504 T699 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 259505 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 259506 T699 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 259507 T699 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 259508 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 259525 T699 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43537/solr
   [junit4]   2> 259526 T699 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 259530 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 259531 T719 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@ed7e0ed8 name:ZooKeeperConnection Watcher:127.0.0.1:43537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 259532 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 259534 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 259544 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 259548 T721 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9b31f5b4 name:ZooKeeperConnection Watcher:127.0.0.1:43537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 259549 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 259552 T699 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 259558 T699 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 259564 T699 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 259567 T699 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:50783_cif%2Fbc
   [junit4]   2> 259569 T699 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:50783_cif%2Fbc
   [junit4]   2> 259576 T699 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 259582 T699 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 259586 T699 oasc.Overseer.start Overseer (id=90144440106549251-127.0.0.1:50783_cif%2Fbc-n_0000000000) starting
   [junit4]   2> 259590 T699 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 259598 T723 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 259599 T699 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 259602 T699 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 259605 T699 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 259610 T722 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 259613 T724 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 259613 T724 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 259623 T724 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 261115 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 261116 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:50783/cif/bc",
   [junit4]   2>  "state":"down",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "node_name":"127.0.0.1:50783_cif%2Fbc",
   [junit4]   2>  "numShards":"1",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"control_collection",
   [junit4]   2>  "shard":null,
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 261117 T722 oasc.Overseer$ClusterStateUpdater.createCollection Create collection control_collection with shards [shard1]
   [junit4]   2> 261117 T722 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 261123 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 261623 T724 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 261624 T724 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/collection1
   [junit4]   2> 261624 T724 oasc.ZkController.createCollectionZkNode Check for collection zkNode:control_collection
   [junit4]   2> 261625 T724 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 261626 T724 oasc.ZkController.readConfigName Load collection config from:/collections/control_collection
   [junit4]   2> 261627 T724 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/collection1/'
   [junit4]   2> 261628 T724 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/collection1/lib/README' to classloader
   [junit4]   2> 261628 T724 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/collection1/lib/classes/' to classloader
   [junit4]   2> 261670 T724 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 261710 T724 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 261712 T724 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 261718 T724 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 262054 T724 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 262055 T724 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 262055 T724 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 262064 T724 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 262068 T724 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 262081 T724 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 262087 T724 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 262093 T724 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 262094 T724 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 262095 T724 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 262095 T724 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 262097 T724 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 262097 T724 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 262098 T724 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 262098 T724 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-controljetty-1375494997035/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/control/data/
   [junit4]   2> 262099 T724 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c2b214
   [junit4]   2> 262100 T724 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/control/data
   [junit4]   2> 262101 T724 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/control/data/index/
   [junit4]   2> 262101 T724 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/control/data/index' doesn't exist. Creating new index...
   [junit4]   2> 262102 T724 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/control/data/index
   [junit4]   2> 262104 T724 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e39307c3 lockFactory=org.apache.lucene.store.NativeFSLockFactory@ba4db974),segFN=segments_1,generation=1}
   [junit4]   2> 262104 T724 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 262106 T724 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 262107 T724 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 262107 T724 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 262108 T724 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 262108 T724 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 262109 T724 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 262109 T724 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 262109 T724 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 262110 T724 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 262111 T724 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 262113 T724 oass.SolrIndexSearcher.<init> Opening Searcher@965735ed main
   [junit4]   2> 262114 T724 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 262114 T724 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 262117 T725 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@965735ed main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 262117 T724 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 262118 T724 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:50783/cif/bc collection:control_collection shard:shard1
   [junit4]   2> 262118 T724 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 262123 T724 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leader_elect/shard1/election
   [junit4]   2> 262128 T724 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 262130 T724 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 262130 T724 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 262131 T724 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:50783/cif/bc/collection1/
   [junit4]   2> 262131 T724 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 262131 T724 oasc.SyncStrategy.syncToMe http://127.0.0.1:50783/cif/bc/collection1/ has no replicas
   [junit4]   2> 262132 T724 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:50783/cif/bc/collection1/ shard1
   [junit4]   2> 262132 T724 oascc.SolrZkClient.makePath makePath: /collections/control_collection/leaders/shard1
   [junit4]   2> 262627 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 262633 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 262638 T724 oasc.ZkController.register We are http://127.0.0.1:50783/cif/bc/collection1/ and leader is http://127.0.0.1:50783/cif/bc/collection1/
   [junit4]   2> 262639 T724 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:50783/cif/bc
   [junit4]   2> 262639 T724 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 262640 T724 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 262640 T724 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 262642 T724 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 262643 T699 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 262644 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 262645 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 262650 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 262654 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 262657 T728 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6b757676 name:ZooKeeperConnection Watcher:127.0.0.1:43537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 262657 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 262659 T699 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 262661 T699 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 262737 T699 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 262745 T699 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:47992
   [junit4]   2> 262746 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 262747 T699 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 262747 T699 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404
   [junit4]   2> 262748 T699 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/'
   [junit4]   2> 262756 T699 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/solr.xml
   [junit4]   2> 262795 T699 oasc.CoreContainer.<init> New CoreContainer -822477237
   [junit4]   2> 262795 T699 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/]
   [junit4]   2> 262797 T699 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 262797 T699 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 262798 T699 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 262799 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 262800 T699 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 262801 T699 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 262802 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 262807 T699 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 262808 T699 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 262809 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 262817 T699 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43537/solr
   [junit4]   2> 262818 T699 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 262821 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 262823 T739 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a8a7292e name:ZooKeeperConnection Watcher:127.0.0.1:43537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 262823 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 262826 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 262836 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 262838 T741 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6fe44af7 name:ZooKeeperConnection Watcher:127.0.0.1:43537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 262838 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 262842 T699 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 263845 T699 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:47992_cif%2Fbc
   [junit4]   2> 263848 T699 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:47992_cif%2Fbc
   [junit4]   2> 263851 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 263852 T741 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 263852 T728 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 263854 T721 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 263860 T742 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 263860 T742 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 263862 T742 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 264138 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 264139 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=1 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:50783/cif/bc",
   [junit4]   2>  "state":"active",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "node_name":"127.0.0.1:50783_cif%2Fbc",
   [junit4]   2>  "numShards":"1",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"control_collection",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 264144 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:47992/cif/bc",
   [junit4]   2>  "state":"down",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "node_name":"127.0.0.1:47992_cif%2Fbc",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"collection1",
   [junit4]   2>  "shard":null,
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 264144 T722 oasc.Overseer$ClusterStateUpdater.createCollection Create collection collection1 with shards [shard1, shard2]
   [junit4]   2> 264145 T722 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 264150 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 264151 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 264151 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 264862 T742 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 264863 T742 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/collection1
   [junit4]   2> 264863 T742 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 264864 T742 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 264865 T742 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 264867 T742 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/collection1/'
   [junit4]   2> 264868 T742 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/collection1/lib/README' to classloader
   [junit4]   2> 264868 T742 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/collection1/lib/classes/' to classloader
   [junit4]   2> 264908 T742 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 264944 T742 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 264946 T742 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 264952 T742 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 265297 T742 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 265298 T742 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 265298 T742 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 265303 T742 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 265306 T742 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 265317 T742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 265322 T742 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 265326 T742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 265327 T742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 265327 T742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 265328 T742 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 265329 T742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 265329 T742 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 265330 T742 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 265330 T742 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty1-1375495000404/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty1/
   [junit4]   2> 265330 T742 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c2b214
   [junit4]   2> 265332 T742 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty1
   [junit4]   2> 265332 T742 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty1/index/
   [junit4]   2> 265332 T742 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty1/index' doesn't exist. Creating new index...
   [junit4]   2> 265333 T742 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty1/index
   [junit4]   2> 265334 T742 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@f2901691 lockFactory=org.apache.lucene.store.NativeFSLockFactory@9c88f160),segFN=segments_1,generation=1}
   [junit4]   2> 265334 T742 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 265337 T742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 265338 T742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 265338 T742 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 265339 T742 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 265339 T742 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 265340 T742 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 265340 T742 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 265341 T742 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 265347 T742 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 265348 T742 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 265350 T742 oass.SolrIndexSearcher.<init> Opening Searcher@177489ba main
   [junit4]   2> 265351 T742 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 265351 T742 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 265355 T743 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@177489ba main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 265356 T742 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 265357 T742 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:47992/cif/bc collection:collection1 shard:shard1
   [junit4]   2> 265357 T742 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 265362 T742 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard1/election
   [junit4]   2> 265367 T742 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 265369 T742 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 265370 T742 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 265370 T742 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:47992/cif/bc/collection1/
   [junit4]   2> 265370 T742 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 265371 T742 oasc.SyncStrategy.syncToMe http://127.0.0.1:47992/cif/bc/collection1/ has no replicas
   [junit4]   2> 265371 T742 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:47992/cif/bc/collection1/ shard1
   [junit4]   2> 265371 T742 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard1
   [junit4]   2> 265659 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 265666 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 265666 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 265666 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 265678 T742 oasc.ZkController.register We are http://127.0.0.1:47992/cif/bc/collection1/ and leader is http://127.0.0.1:47992/cif/bc/collection1/
   [junit4]   2> 265678 T742 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:47992/cif/bc
   [junit4]   2> 265678 T742 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 265679 T742 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 265679 T742 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 265681 T742 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 265682 T699 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 265683 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 265684 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 265775 T699 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 265780 T699 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38311
   [junit4]   2> 265781 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 265782 T699 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 265783 T699 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433
   [junit4]   2> 265784 T699 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/'
   [junit4]   2> 265792 T699 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/solr.xml
   [junit4]   2> 265827 T699 oasc.CoreContainer.<init> New CoreContainer 214010815
   [junit4]   2> 265828 T699 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/]
   [junit4]   2> 265829 T699 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 265830 T699 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 265830 T699 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 265831 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 265832 T699 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 265832 T699 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 265833 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 265834 T699 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 265834 T699 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 265835 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 265840 T699 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43537/solr
   [junit4]   2> 265841 T699 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 265845 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 265847 T755 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6f6bd00 name:ZooKeeperConnection Watcher:127.0.0.1:43537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 265848 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 265849 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 265857 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 265858 T757 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@c5e26582 name:ZooKeeperConnection Watcher:127.0.0.1:43537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 265859 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 265862 T699 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 266865 T699 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38311_cif%2Fbc
   [junit4]   2> 266866 T699 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38311_cif%2Fbc
   [junit4]   2> 266868 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 266868 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 266868 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 2)
   [junit4]   2> 266869 T757 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 266871 T728 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 266871 T721 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 266872 T741 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 266878 T758 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 266879 T758 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 266885 T758 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 267172 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 267173 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:47992/cif/bc",
   [junit4]   2>  "state":"active",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "node_name":"127.0.0.1:47992_cif%2Fbc",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"collection1",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 267176 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:38311/cif/bc",
   [junit4]   2>  "state":"down",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node2",
   [junit4]   2>  "node_name":"127.0.0.1:38311_cif%2Fbc",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"collection1",
   [junit4]   2>  "shard":null,
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 267177 T722 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 267177 T722 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard2
   [junit4]   2> 267181 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 267181 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 267181 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 267181 T757 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 267886 T758 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 267887 T758 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/collection1
   [junit4]   2> 267887 T758 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 267888 T758 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 267888 T758 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 267889 T758 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/collection1/'
   [junit4]   2> 267890 T758 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/collection1/lib/README' to classloader
   [junit4]   2> 267890 T758 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/collection1/lib/classes/' to classloader
   [junit4]   2> 267925 T758 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 267974 T758 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 267980 T758 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 267991 T758 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 268486 T758 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 268487 T758 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 268487 T758 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 268496 T758 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 268500 T758 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 268513 T758 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 268518 T758 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 268522 T758 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 268523 T758 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 268523 T758 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 268524 T758 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 268525 T758 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 268525 T758 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 268525 T758 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 268526 T758 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty2-1375495003433/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty2/
   [junit4]   2> 268526 T758 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c2b214
   [junit4]   2> 268527 T758 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty2
   [junit4]   2> 268527 T758 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty2/index/
   [junit4]   2> 268528 T758 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty2/index' doesn't exist. Creating new index...
   [junit4]   2> 268528 T758 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty2/index
   [junit4]   2> 268530 T758 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@b39ac42d lockFactory=org.apache.lucene.store.NativeFSLockFactory@fe1f542f),segFN=segments_1,generation=1}
   [junit4]   2> 268530 T758 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 268533 T758 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 268533 T758 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 268534 T758 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 268534 T758 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 268535 T758 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 268535 T758 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 268535 T758 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 268536 T758 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 268536 T758 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 268538 T758 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 268540 T758 oass.SolrIndexSearcher.<init> Opening Searcher@dbc4d90c main
   [junit4]   2> 268541 T758 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 268541 T758 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 268545 T759 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@dbc4d90c main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 268546 T758 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 268547 T758 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:38311/cif/bc collection:collection1 shard:shard2
   [junit4]   2> 268547 T758 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 268552 T758 oascc.SolrZkClient.makePath makePath: /collections/collection1/leader_elect/shard2/election
   [junit4]   2> 268556 T758 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 268558 T758 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 268559 T758 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 268559 T758 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:38311/cif/bc/collection1/
   [junit4]   2> 268559 T758 oasc.SyncStrategy.syncReplicas Sync Success - now sync replicas to me
   [junit4]   2> 268560 T758 oasc.SyncStrategy.syncToMe http://127.0.0.1:38311/cif/bc/collection1/ has no replicas
   [junit4]   2> 268560 T758 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:38311/cif/bc/collection1/ shard2
   [junit4]   2> 268561 T758 oascc.SolrZkClient.makePath makePath: /collections/collection1/leaders/shard2
   [junit4]   2> 268687 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 268693 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 268694 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 268694 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 268696 T757 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 268715 T758 oasc.ZkController.register We are http://127.0.0.1:38311/cif/bc/collection1/ and leader is http://127.0.0.1:38311/cif/bc/collection1/
   [junit4]   2> 268716 T758 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:38311/cif/bc
   [junit4]   2> 268716 T758 oasc.ZkController.checkRecovery I am the leader, no recovery necessary
   [junit4]   2> 268717 T758 oasc.ZkController.publish publishing core=collection1 state=active
   [junit4]   2> 268717 T758 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 268720 T758 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 268722 T699 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 268723 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 268724 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 268817 T699 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 268824 T699 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:42474
   [junit4]   2> 268825 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 268826 T699 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 268826 T699 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474
   [junit4]   2> 268827 T699 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/'
   [junit4]   2> 268835 T699 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/solr.xml
   [junit4]   2> 268868 T699 oasc.CoreContainer.<init> New CoreContainer -706107051
   [junit4]   2> 268869 T699 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/]
   [junit4]   2> 268870 T699 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 268871 T699 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 268871 T699 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 268872 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 268873 T699 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 268874 T699 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 268874 T699 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 268875 T699 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 268876 T699 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 268876 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 268882 T699 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:43537/solr
   [junit4]   2> 268883 T699 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 268886 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268888 T771 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@41b6c76d name:ZooKeeperConnection Watcher:127.0.0.1:43537 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268888 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268893 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 268901 T699 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 268902 T773 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@6971bf0a name:ZooKeeperConnection Watcher:127.0.0.1:43537/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 268903 T699 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 268906 T699 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 269910 T699 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:42474_cif%2Fbc
   [junit4]   2> 269911 T699 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:42474_cif%2Fbc
   [junit4]   2> 269914 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 269914 T757 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 269914 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 269914 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 3)
   [junit4]   2> 269918 T773 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 269919 T741 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 269919 T757 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 269919 T721 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 269919 T728 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 269927 T774 oasc.ZkController.publish publishing core=collection1 state=down
   [junit4]   2> 269928 T774 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 269931 T774 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 270200 T722 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 270202 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:38311/cif/bc",
   [junit4]   2>  "state":"active",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node2",
   [junit4]   2>  "node_name":"127.0.0.1:38311_cif%2Fbc",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"collection1",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 270206 T722 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "base_url":"http://127.0.0.1:42474/cif/bc",
   [junit4]   2>  "state":"down",
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node3",
   [junit4]   2>  "node_name":"127.0.0.1:42474_cif%2Fbc",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "roles":null,
   [junit4]   2>  "collection":"collection1",
   [junit4]   2>  "shard":null,
   [junit4]   2>  "core":"collection1"}
   [junit4]   2> 270206 T722 oasc.Overseer$ClusterStateUpdater.updateState Collection already exists with numShards=2
   [junit4]   2> 270206 T722 oasc.Overseer$ClusterStateUpdater.updateState Assigning new node to shard shard=shard1
   [junit4]   2> 270211 T773 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 270211 T728 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 270211 T757 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 270212 T721 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 270212 T741 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
   [junit4]   2> 270932 T774 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for collection1
   [junit4]   2> 270932 T774 oasc.CoreContainer.create Creating SolrCore 'collection1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/collection1
   [junit4]   2> 270933 T774 oasc.ZkController.createCollectionZkNode Check for collection zkNode:collection1
   [junit4]   2> 270934 T774 oasc.ZkController.createCollectionZkNode Collection zkNode exists
   [junit4]   2> 270934 T774 oasc.ZkController.readConfigName Load collection config from:/collections/collection1
   [junit4]   2> 270936 T774 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/collection1/'
   [junit4]   2> 270936 T774 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/collection1/lib/README' to classloader
   [junit4]   2> 270937 T774 oasc.SolrResourceLoader.replaceClassLoader Adding 'file:/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/collection1/lib/classes/' to classloader
   [junit4]   2> 270977 T774 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 271027 T774 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 271029 T774 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 271038 T774 oass.IndexSchema.readSchema [collection1] Schema name=test
   [junit4]   2> 271615 T774 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 271616 T774 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 271619 T774 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 271637 T774 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 271641 T774 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 271668 T774 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 271674 T774 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 271680 T774 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 271681 T774 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 271682 T774 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 271682 T774 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 271683 T774 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 271684 T774 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 271686 T774 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 271687 T774 oasc.SolrCore.<init> [collection1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty3-1375495006474/collection1/, dataDir=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty3/
   [junit4]   2> 271687 T774 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@37c2b214
   [junit4]   2> 271688 T774 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty3
   [junit4]   2> 271689 T774 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty3/index/
   [junit4]   2> 271689 T774 oasc.SolrCore.initIndex WARN [collection1] Solr index directory './org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty3/index' doesn't exist. Creating new index...
   [junit4]   2> 271690 T774 oasc.CachingDirectoryFactory.get return new directory for ./org.apache.solr.cloud.BasicDistributedZk2Test-1375494996737/jetty3/index
   [junit4]   2> 271692 T774 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@ed9594ce lockFactory=org.apache.lucene.store.NativeFSLockFactory@3d0c0b06),segFN=segments_1,generation=1}
   [junit4]   2> 271692 T774 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 271696 T774 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 271697 T774 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 271698 T774 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 271699 T774 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 271700 T774 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 271700 T774 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 271701 T774 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 271702 T774 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 271702 T774 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 271704 T774 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 271707 T774 oass.SolrIndexSearcher.<init> Opening Searcher@876c4bff main
   [junit4]   2> 271708 T774 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 271709 T774 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 271719 T775 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@876c4bff main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 271725 T774 oasc.CoreContainer.registerCore registering core: collection1
   [junit4]   2> 271726 T774 oasc.ZkController.register Register replica - core:collection1 address:http://127.0.0.1:42474/cif/bc collection:collection1 shard:shard1
   [junit4]   2> 271726 T774 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 271743 T774 oasc.ZkController.register We are http://127.0.0.1:42474/cif/bc/collection1/ and leader is http://127.0.0.1:47992/cif/bc/collection1/
   [junit4]   2> 271743 T774 oasc.ZkController.register No LogReplay needed for core=collection1 baseURL=http://127.0.0.1:42474/cif/bc
   [junit4]   2> 271744 T774 oasc.ZkController.checkRecovery Core needs to recover:collection1
   [junit4]   2> 271744 T774 oasu.DefaultSolrCoreState.doRecovery Running recovery - first canceling any ongoing recovery
   [junit4]   2> 271745 T774 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 271746 T699 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1
   [junit4]   2> 271747 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 271748 T699 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> ASYNC  NEW_CORE C439 name=collection1 org.apache.solr.core.SolrCore@1f4fbf1c url=http://127.0.0.1:42474/cif/bc/collection1 node=127.0.0.1:42474_cif%2Fbc C439_STATE=coll:collection1 core:collection1 props:{base_url=http://127.0.0.1:42474/cif/bc, state=down, node_name=127.0.0.1:42474_cif%2Fbc, core=collection1}
   [junit4]   2> 271755 T776 C439 P42474 oasc.RecoveryStrategy.run Starting recovery process.  core=collection1 recoveringAfterStartup=true
   [junit4]   2> 271756 T776 C439 P42474 oasc.RecoveryStrategy.doRecovery ###### startupVersions=[]
   [junit4]   2> 271760 T776 C439 P42474 oasc.ZkController.publish publishing core=collection1 state=recovering
   [junit4]   2> 271760 T776 C439 P42474 oasc.ZkController.publish numShards not found on descriptor - reading it from system property
   [junit4]   2> 271764 T776 C439 P42474 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 271780 T736 oasha.CoreAdminHandler.handleWaitForStateAction Going to wait for coreNodeName: core_node3, state: recovering, checkLive: true, onlyIfLeader: true
   [junit4]   2> 271868 T699 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 271875 T699 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:58557
   [junit4]   2> 271877 T699 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 271878 T699 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 271878 T699 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1375495009499
   [junit4]   2> 271879 T699 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1375495009499/'
   [junit4]   2> 271891 T699 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1375495009499/solr.xml
   [junit4]   2> 271942 T699 oasc.CoreContainer.<init> New CoreContainer 1969694248
   [junit4]   2> 271943 T699 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J1/./org.apache.solr.cloud.BasicDistributedZk2Test-jetty4-1375495009499/]
   [junit4]   2> 271945 T699 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 271946 T699 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 27

[...truncated too long message...]

04D47D73F71 -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_SD -Dtests.timezone=AGT -Dtests.file.encoding=UTF-8
   [junit4] ERROR    138s J1 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:56594
   [junit4]    > at __randomizedtesting.SeedInfo.seed([9790304D47D73F71:1676BE5530885F4D]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1535)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:677)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > at java.lang.Thread.run(Thread.java:780)
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:56594 refused
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
   [junit4]    > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    > ... 45 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:369)
   [junit4]    > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:230)
   [junit4]    > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:212)
   [junit4]    > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:402)
   [junit4]    > at java.net.Socket.connect(Socket.java:590)
   [junit4]    > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    > ... 52 more
   [junit4]   2> 799704 T2038 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 138043 T2037 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 800904 T2120 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/new_implicit_coll_0/leader_elect/c/election
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2>
   [junit4]   2> 800904 T2120 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 800905 T2120 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 800905 T2120 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 800905 T2120 oasc.LeaderElector$1.process WARN  org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /collections/new_implicit_coll_0/leader_elect/b/election
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:127)
   [junit4]   2> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
   [junit4]   2> at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1468)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:235)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient$6.execute(SolrZkClient.java:232)
   [junit4]   2> at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
   [junit4]   2> at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:232)
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:84)
   [junit4]   2> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
   [junit4]   2> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
   [junit4]   2> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
   [junit4]   2>
   [junit4]   2> NOTE: test params are: codec=Appending, sim=RandomSimilarityProvider(queryNorm=false,coord=no): {}, locale=ar_SD, timezone=AGT
   [junit4]   2> NOTE: Linux 3.2.0-51-generic x86/IBM Corporation 1.7.0 (32-bit)/cpus=8,threads=1,free=84532896,total=292487168
   [junit4]   2> NOTE: All tests run in this JVM: [TestArbitraryIndexDir, StressHdfsTest, ConvertedLegacyTest, StatsComponentTest, TestJmxMonitoredMap, NumericFieldsTest, TestRecovery, TestCloudManagedSchemaAddField, TestHashPartitioner, HardAutoCommitTest, HighlighterTest, TestSuggestSpellingConverter, TestAddFieldRealTimeGet, XsltUpdateRequestHandlerTest, TestIndexingPerformance, HdfsDirectoryTest, OutputWriterTest, SimplePostToolTest, TestCollationField, TestRandomFaceting, TestCodecSupport, SpellCheckComponentTest, TestDFRSimilarityFactory, URLClassifyProcessorTest, DebugComponentTest, TestZkChroot, FieldAnalysisRequestHandlerTest, HighlighterConfigTest, TestSolrQueryParserResource, TestMaxScoreQueryParser, DirectUpdateHandlerTest, TestFoldingMultitermQuery, IndexReaderFactoryTest, ClusterStateTest, BasicDistributedZk2Test, IndexSchemaRuntimeFieldTest, SpellCheckCollatorTest, BlockDirectoryTest, HdfsUnloadDistributedZkTest, CurrencyFieldXmlFileTest, CopyFieldTest, DirectSolrSpellCheckerTest, TestCollationKeyRangeQueries, TestPostingsSolrHighlighter, FullSolrCloudDistribCmdsTest, NotRequiredUniqueKeyTest, TestWordDelimiterFilterFactory, ChaosMonkeySafeLeaderTest, CurrencyFieldOpenExchangeTest, TestFieldTypeCollectionResource, NoCacheHeaderTest, CoreContainerCoreInitFailuresTest, TestIndexSearcher, IndexBasedSpellCheckerTest, TestShardHandlerFactory, OpenExchangeRatesOrgProviderTest, TestDynamicFieldResource, TestCSVResponseWriter, SuggesterWFSTTest, TestOmitPositions, LeaderElectionIntegrationTest, BlockCacheTest, TestMergePolicyConfig, DocValuesMultiTest, TestRangeQuery, TestLMJelinekMercerSimilarityFactory, TestSolrXMLSerializer, JSONWriterTest, TestFastOutputStream, QueryResultKeyTest, MultiTermTest, SolrCoreCheckLockOnStartupTest, SolrTestCaseJ4Test, TestClassNameShortening, TermVectorComponentDistributedTest, TestFieldResource, UnloadDistributedZkTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J1 in 139.32s, 1 test, 1 error <<< FAILURES!

[...truncated 529 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:388: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: There were test failures: 321 suites, 1352 tests, 2 errors, 34 ignored (6 assumptions)

Total time: 47 minutes 45 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/ibm-j9-jdk7 -Xjit:exclude={org/apache/lucene/util/fst/FST.pack(IIF)Lorg/apache/lucene/util/fst/FST;}
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

[JENKINS] Lucene-Solr-4.x-Linux (32bit/jdk1.6.0_45) - Build # 6776 - Still Failing!

Policeman Jenkins Server-2
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-4.x-Linux/6776/
Java: 32bit/jdk1.6.0_45 -server -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testDistribSearch

Error Message:
Server refused connection at: http://127.0.0.1:54076

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:54076
        at __randomizedtesting.SeedInfo.seed([28BC2E40482BEFCE:A95AA0583F748FF2]:0)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
        at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1535)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:677)
        at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
        at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1559)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.access$600(RandomizedRunner.java:79)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:737)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:773)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:787)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
        at org.apache.lucene.util.TestRuleFieldCacheSanity$1.evaluate(TestRuleFieldCacheSanity.java:51)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:782)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:442)
        at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:746)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:648)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:682)
        at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:693)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:53)
        at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
        at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
        at com.carrotsearch.randomizedtesting.rules.SystemPropertiesInvariantRule$1.evaluate(SystemPropertiesInvariantRule.java:55)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:43)
        at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
        at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:70)
        at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:358)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:54076 refused
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
        at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
        at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
        at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
        at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
        ... 45 more
Caused by: java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
        at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
        ... 52 more




Build Log:
[...truncated 9243 lines...]
   [junit4] Suite: org.apache.solr.cloud.CollectionsAPIDistributedZkTest
   [junit4]   2> 751453 T2241 oas.BaseDistributedSearchTestCase.initHostContext Setting hostContext system property: /
   [junit4]   2> 751458 T2241 oas.SolrTestCaseJ4.setUp ###Starting testDistribSearch
   [junit4]   2> Creating dataDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./solrtest-CollectionsAPIDistributedZkTest-1375500973844
   [junit4]   2> 751458 T2241 oasc.ZkTestServer.run STARTING ZK TEST SERVER
   [junit4]   2> 751459 T2242 oasc.ZkTestServer$ZKServerMain.runFromConfig Starting server
   [junit4]   2> 751559 T2241 oasc.ZkTestServer.run start zk server on port:41165
   [junit4]   2> 751560 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 751625 T2248 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@141815c name:ZooKeeperConnection Watcher:127.0.0.1:41165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 751625 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 751625 T2241 oascc.SolrZkClient.makePath makePath: /solr
   [junit4]   2> 751630 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 751631 T2250 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1ed8fa6 name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 751631 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 751632 T2241 oascc.SolrZkClient.makePath makePath: /collections/collection1
   [junit4]   2> 751635 T2241 oascc.SolrZkClient.makePath makePath: /collections/collection1/shards
   [junit4]   2> 751639 T2241 oascc.SolrZkClient.makePath makePath: /collections/control_collection
   [junit4]   2> 751642 T2241 oascc.SolrZkClient.makePath makePath: /collections/control_collection/shards
   [junit4]   2> 751646 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/solrconfig-tlog.xml to /configs/conf1/solrconfig.xml
   [junit4]   2> 751646 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/solrconfig.xml
   [junit4]   2> 751652 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/schema.xml to /configs/conf1/schema.xml
   [junit4]   2> 751653 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/schema.xml
   [junit4]   2> 751657 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/stopwords.txt to /configs/conf1/stopwords.txt
   [junit4]   2> 751657 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/stopwords.txt
   [junit4]   2> 751661 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/protwords.txt to /configs/conf1/protwords.txt
   [junit4]   2> 751662 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/protwords.txt
   [junit4]   2> 751665 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/currency.xml to /configs/conf1/currency.xml
   [junit4]   2> 751666 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/currency.xml
   [junit4]   2> 751669 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/open-exchange-rates.json to /configs/conf1/open-exchange-rates.json
   [junit4]   2> 751670 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/open-exchange-rates.json
   [junit4]   2> 751674 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/mapping-ISOLatin1Accent.txt to /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 751674 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/mapping-ISOLatin1Accent.txt
   [junit4]   2> 751678 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/old_synonyms.txt to /configs/conf1/old_synonyms.txt
   [junit4]   2> 751678 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/old_synonyms.txt
   [junit4]   2> 751682 T2241 oasc.AbstractZkTestCase.putConfig put /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test-files/solr/collection1/conf/synonyms.txt to /configs/conf1/synonyms.txt
   [junit4]   2> 751682 T2241 oascc.SolrZkClient.makePath makePath: /configs/conf1/synonyms.txt
   [junit4]   2> 751769 T2241 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 751772 T2241 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:44838
   [junit4]   2> 751772 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 751773 T2241 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 751773 T2241 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375500974074
   [junit4]   2> 751773 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375500974074/'
   [junit4]   2> 751785 T2241 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375500974074/solr.xml
   [junit4]   2> 751820 T2241 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 751820 T2241 oasc.CoreContainer.<init> New CoreContainer 5934438
   [junit4]   2> 751821 T2241 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375500974074/]
   [junit4]   2> 751822 T2241 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 751822 T2241 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 751823 T2241 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 751823 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 751823 T2241 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 751824 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 751824 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 751824 T2241 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 751825 T2241 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 751825 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 751828 T2241 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41165/solr
   [junit4]   2> 751829 T2241 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 751829 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 751831 T2261 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@f94bba name:ZooKeeperConnection Watcher:127.0.0.1:41165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 751831 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 751833 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 751835 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 751837 T2263 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a07606 name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 751837 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 751839 T2241 oascc.SolrZkClient.makePath makePath: /overseer/queue
   [junit4]   2> 751843 T2241 oascc.SolrZkClient.makePath makePath: /overseer/collection-queue-work
   [junit4]   2> 751847 T2241 oascc.SolrZkClient.makePath makePath: /live_nodes
   [junit4]   2> 751850 T2241 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:44838_
   [junit4]   2> 751851 T2241 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:44838_
   [junit4]   2> 751856 T2241 oascc.SolrZkClient.makePath makePath: /overseer_elect/election
   [junit4]   2> 751863 T2241 oascc.SolrZkClient.makePath makePath: /overseer_elect/leader
   [junit4]   2> 751866 T2241 oasc.Overseer.start Overseer (id=90144831822036995-127.0.0.1:44838_-n_0000000000) starting
   [junit4]   2> 751869 T2241 oascc.SolrZkClient.makePath makePath: /overseer/queue-work
   [junit4]   2> 751874 T2265 oasc.OverseerCollectionProcessor.run Process current queue of collection creations
   [junit4]   2> 751875 T2241 oascc.SolrZkClient.makePath makePath: /clusterstate.json
   [junit4]   2> 751879 T2241 oascc.SolrZkClient.makePath makePath: /aliases.json
   [junit4]   2> 751881 T2241 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 751885 T2264 oasc.Overseer$ClusterStateUpdater.run Starting to work on the main queue
   [junit4]   2> 751887 T2241 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375500974074
   [junit4]   2> 751914 T2241 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-controljetty-1375500974074/conf/
   [junit4]   2> 751914 T2241 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 751915 T2241 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 751915 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 751915 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 751919 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:
   [junit4]   2> 751920 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 751921 T2268 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@a24a0e name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 751921 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 751923 T2241 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 751926 T2241 oasc.ChaosMonkey.monkeyLog monkey: init - expire sessions:true cause connection loss:true
   [junit4]   2> 752001 T2241 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 752003 T2241 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:45099
   [junit4]   2> 752004 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 752004 T2241 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 752005 T2241 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313
   [junit4]   2> 752005 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/'
   [junit4]   2> 752017 T2241 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/solr.xml
   [junit4]   2> 752052 T2241 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 752053 T2241 oasc.CoreContainer.<init> New CoreContainer 5367096
   [junit4]   2> 752053 T2241 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/]
   [junit4]   2> 752054 T2241 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 752055 T2241 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 752055 T2241 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 752055 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 752056 T2241 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 752056 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 752056 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 752057 T2241 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 752057 T2241 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 752058 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 752060 T2241 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41165/solr
   [junit4]   2> 752061 T2241 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 752062 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 752063 T2279 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@9fc2c5 name:ZooKeeperConnection Watcher:127.0.0.1:41165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 752063 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 752065 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 752068 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 752080 T2281 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@1f5d09b name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 752080 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 752085 T2241 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 753089 T2241 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:45099_
   [junit4]   2> 753091 T2241 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:45099_
   [junit4]   2> 753096 T2281 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 753097 T2263 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 753097 T2268 oascc.ZkStateReader$3.process Updating live nodes... (2)
   [junit4]   2> 753103 T2241 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313
   [junit4]   2> 753127 T2241 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/conf/
   [junit4]   2> 753128 T2241 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 753128 T2241 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 753129 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 753129 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 753220 T2241 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 753222 T2241 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:54076
   [junit4]   2> 753223 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 753223 T2241 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 753223 T2241 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518
   [junit4]   2> 753224 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/'
   [junit4]   2> 753234 T2241 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/solr.xml
   [junit4]   2> 753267 T2241 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 753267 T2241 oasc.CoreContainer.<init> New CoreContainer 13826504
   [junit4]   2> 753268 T2241 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/]
   [junit4]   2> 753269 T2241 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 753269 T2241 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 753269 T2241 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 753270 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 753270 T2241 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 753270 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 753271 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 753271 T2241 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 753271 T2241 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 753272 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 753275 T2241 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41165/solr
   [junit4]   2> 753275 T2241 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 753276 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753277 T2293 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@10ce600 name:ZooKeeperConnection Watcher:127.0.0.1:41165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753277 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 753279 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 753282 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 753283 T2295 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@90dd82 name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 753283 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 753288 T2241 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 754293 T2241 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:54076_
   [junit4]   2> 754294 T2241 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:54076_
   [junit4]   2> 754299 T2281 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 754299 T2263 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 754300 T2295 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 754300 T2268 oascc.ZkStateReader$3.process Updating live nodes... (3)
   [junit4]   2> 754307 T2241 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518
   [junit4]   2> 754334 T2241 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/conf/
   [junit4]   2> 754335 T2241 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 754335 T2241 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 754335 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 754336 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 754406 T2241 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 754408 T2241 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:38163
   [junit4]   2> 754408 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 754409 T2241 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 754409 T2241 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724
   [junit4]   2> 754410 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/'
   [junit4]   2> 754420 T2241 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/solr.xml
   [junit4]   2> 754453 T2241 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 754453 T2241 oasc.CoreContainer.<init> New CoreContainer 27206436
   [junit4]   2> 754453 T2241 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/]
   [junit4]   2> 754455 T2241 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 754455 T2241 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 754455 T2241 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 754456 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 754456 T2241 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 754456 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 754457 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 754457 T2241 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 754457 T2241 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 754458 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 754460 T2241 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41165/solr
   [junit4]   2> 754461 T2241 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 754462 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 754463 T2307 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@bc49bc name:ZooKeeperConnection Watcher:127.0.0.1:41165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 754463 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 754465 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 754468 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 754469 T2309 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@17d7fd9 name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 754469 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 754474 T2241 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 755479 T2241 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:38163_
   [junit4]   2> 755480 T2241 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:38163_
   [junit4]   2> 755488 T2295 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 755488 T2263 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 755489 T2309 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 755490 T2268 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 755490 T2281 oascc.ZkStateReader$3.process Updating live nodes... (4)
   [junit4]   2> 755498 T2241 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724
   [junit4]   2> 755522 T2241 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/conf/
   [junit4]   2> 755523 T2241 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 755523 T2241 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 755524 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 755524 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 755601 T2241 oejs.Server.doStart jetty-8.1.10.v20130312
   [junit4]   2> 755602 T2241 oejs.AbstractConnector.doStart Started SelectChannelConnector@127.0.0.1:37275
   [junit4]   2> 755603 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init()
   [junit4]   2> 755604 T2241 oasc.SolrResourceLoader.locateSolrHome JNDI not configured for solr (NoInitialContextEx)
   [junit4]   2> 755604 T2241 oasc.SolrResourceLoader.locateSolrHome using system property solr.solr.home: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913
   [junit4]   2> 755604 T2241 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/'
   [junit4]   2> 755615 T2241 oasc.ConfigSolr.fromFile Loading container configuration from /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/solr.xml
   [junit4]   2> 755647 T2241 oasc.ConfigSolrXml.<init> Config-defined core root directory:
   [junit4]   2> 755647 T2241 oasc.CoreContainer.<init> New CoreContainer 16375865
   [junit4]   2> 755647 T2241 oasc.CoreContainer.load Loading cores into CoreContainer [instanceDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/]
   [junit4]   2> 755649 T2241 oashc.HttpShardHandlerFactory.getParameter Setting socketTimeout to: 120000
   [junit4]   2> 755649 T2241 oashc.HttpShardHandlerFactory.getParameter Setting urlScheme to: http://
   [junit4]   2> 755649 T2241 oashc.HttpShardHandlerFactory.getParameter Setting connTimeout to: 15000
   [junit4]   2> 755649 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxConnectionsPerHost to: 20
   [junit4]   2> 755650 T2241 oashc.HttpShardHandlerFactory.getParameter Setting corePoolSize to: 0
   [junit4]   2> 755650 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maximumPoolSize to: 2147483647
   [junit4]   2> 755650 T2241 oashc.HttpShardHandlerFactory.getParameter Setting maxThreadIdleTime to: 5
   [junit4]   2> 755651 T2241 oashc.HttpShardHandlerFactory.getParameter Setting sizeOfQueue to: -1
   [junit4]   2> 755651 T2241 oashc.HttpShardHandlerFactory.getParameter Setting fairnessPolicy to: false
   [junit4]   2> 755651 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnectionsPerHost=20&maxConnections=10000&socketTimeout=120000&connTimeout=15000&retry=false
   [junit4]   2> 755654 T2241 oasc.ZkContainer.initZooKeeper Zookeeper client=127.0.0.1:41165/solr
   [junit4]   2> 755655 T2241 oasc.ZkController.checkChrootPath zkHost includes chroot
   [junit4]   2> 755655 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 755657 T2321 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@efc3d0 name:ZooKeeperConnection Watcher:127.0.0.1:41165 got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 755657 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 755659 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=500&maxConnectionsPerHost=16&socketTimeout=120000&connTimeout=15000
   [junit4]   2> 755662 T2241 oascc.ConnectionManager.waitForConnected Waiting for client to connect to ZooKeeper
   [junit4]   2> 755663 T2323 oascc.ConnectionManager.process Watcher org.apache.solr.common.cloud.ConnectionManager@d44dc8 name:ZooKeeperConnection Watcher:127.0.0.1:41165/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 755663 T2241 oascc.ConnectionManager.waitForConnected Client is connected to ZooKeeper
   [junit4]   2> 755668 T2241 oascc.ZkStateReader.createClusterStateWatchersAndUpdate Updating cluster state from ZooKeeper...
   [junit4]   2> 756672 T2241 oasc.ZkController.createEphemeralLiveNode Register node as live in ZooKeeper:/live_nodes/127.0.0.1:37275_
   [junit4]   2> 756675 T2241 oascc.SolrZkClient.makePath makePath: /live_nodes/127.0.0.1:37275_
   [junit4]   2> 756681 T2295 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 756682 T2268 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 756682 T2309 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 756682 T2323 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 756683 T2263 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 756683 T2281 oascc.ZkStateReader$3.process Updating live nodes... (5)
   [junit4]   2> 756690 T2241 oasc.CorePropertiesLocator.discover Looking for core definitions underneath /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913
   [junit4]   2> 756715 T2241 oasc.CorePropertiesLocator.discoverUnder Found core conf in /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/conf/
   [junit4]   2> 756716 T2241 oasc.CorePropertiesLocator.discover Found 1 core definitions
   [junit4]   2> 756717 T2241 oass.SolrDispatchFilter.init user.dir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0
   [junit4]   2> 756717 T2241 oass.SolrDispatchFilter.init SolrDispatchFilter.init() done
   [junit4]   2> 756717 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 756720 T2241 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=128&maxConnectionsPerHost=32&followRedirects=false
   [junit4]   2> 756723 T2275 oasha.CollectionsHandler.handleCreateAction Creating Collection : numShards=2&name=nodes_used_collection&replicationFactor=2&action=CREATE&wt=javabin&version=2
   [junit4]   2> 756724 T2263 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 756726 T2263 oasc.DistributedQueue$LatchChildWatcher.process Watcher fired on path: /overseer/collection-queue-work state: SyncConnected type NodeChildrenChanged
   [junit4]   2> 756727 T2265 oasc.OverseerCollectionProcessor.run Overseer Collection Processor: Get the message id:/overseer/collection-queue-work/qn-0000000000 message:{
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 756727 T2265 oasc.OverseerCollectionProcessor.processMessage WARN OverseerCollectionProcessor.processMessage : createcollection , {
   [junit4]   2>  "operation":"createcollection",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "name":"nodes_used_collection",
   [junit4]   2>  "replicationFactor":"2"}
   [junit4]   2> 757898 T2264 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 757899 T2264 oasc.Overseer$ClusterStateUpdater.buildCollection building a new collection: nodes_used_collection
   [junit4]   2> 757899 T2264 oasc.Overseer$ClusterStateUpdater.createCollection Create collection nodes_used_collection with shards [shard1, shard2]
   [junit4]   2> 757905 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 757905 T2309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 757905 T2281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 757905 T2263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 757905 T2323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 757906 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 757933 T2265 oasc.OverseerCollectionProcessor.createCollection going to create cores replicas shardNames [shard1, shard2] , repFactor : 2
   [junit4]   2> 757934 T2265 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica1 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:45099_
   [junit4]   2> 757934 T2265 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard1_replica2 as part of slice shard1 of collection nodes_used_collection on 127.0.0.1:37275_
   [junit4]   2> 757934 T2265 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica1 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:54076_
   [junit4]   2> 757935 T2265 oasc.OverseerCollectionProcessor.createCollection Creating shard nodes_used_collection_shard2_replica2 as part of slice shard2 of collection nodes_used_collection on 127.0.0.1:38163_
   [junit4]   2> 757937 T2319 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica2 state=down
   [junit4]   2> 757937 T2303 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica2 state=down
   [junit4]   2> 757937 T2290 oasc.ZkController.publish publishing core=nodes_used_collection_shard2_replica1 state=down
   [junit4]   2> 757937 T2277 oasc.ZkController.publish publishing core=nodes_used_collection_shard1_replica1 state=down
   [junit4]   2> 757939 T2319 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 757940 T2277 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 757940 T2290 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 757940 T2303 oasc.ZkController.waitForCoreNodeName look for our core node name
   [junit4]   2> 759418 T2264 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 759419 T2264 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 759420 T2264 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node1",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:37275_",
   [junit4]   2>  "base_url":"http://127.0.0.1:37275"}
   [junit4]   2> 759425 T2264 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 759425 T2264 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node2",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica2",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:38163_",
   [junit4]   2>  "base_url":"http://127.0.0.1:38163"}
   [junit4]   2> 759430 T2264 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 759430 T2264 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node3",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard2",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard2_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:54076_",
   [junit4]   2>  "base_url":"http://127.0.0.1:54076"}
   [junit4]   2> 759436 T2264 oasc.Overseer$ClusterStateUpdater.updateState WARN Could not find collection node for nodes_used_collection, skipping publish state
   [junit4]   2> 759436 T2264 oasc.Overseer$ClusterStateUpdater.updateState Update state numShards=2 message={
   [junit4]   2>  "operation":"state",
   [junit4]   2>  "core_node_name":"core_node4",
   [junit4]   2>  "numShards":"2",
   [junit4]   2>  "shard":"shard1",
   [junit4]   2>  "roles":null,
   [junit4]   2>  "state":"down",
   [junit4]   2>  "shard_state":"active",
   [junit4]   2>  "core":"nodes_used_collection_shard1_replica1",
   [junit4]   2>  "collection":"nodes_used_collection",
   [junit4]   2>  "shard_range":null,
   [junit4]   2>  "node_name":"127.0.0.1:45099_",
   [junit4]   2>  "base_url":"http://127.0.0.1:45099"}
   [junit4]   2> 759440 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 759440 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 759441 T2263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 759441 T2281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 759441 T2309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 759441 T2323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 759940 T2277 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica1
   [junit4]   2> 759941 T2290 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica1
   [junit4]   2> 759942 T2319 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard1_replica2
   [junit4]   2> 759943 T2290 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/nodes_used_collection_shard2_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/nodes_used_collection_shard2_replica1/core.properties (No such file or directory)
   [junit4]   2> 759942 T2277 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/nodes_used_collection_shard1_replica1/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/nodes_used_collection_shard1_replica1/core.properties (No such file or directory)
   [junit4]   2> 759942 T2303 oasc.ZkController.waitForShardId waiting to find shard id in clusterstate for nodes_used_collection_shard2_replica2
   [junit4]   2> 759944 T2277 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/nodes_used_collection_shard1_replica1
   [junit4]   2> 759945 T2277 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 759943 T2290 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica1' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/nodes_used_collection_shard2_replica1
   [junit4]   2> 759946 T2290 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 759943 T2319 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/nodes_used_collection_shard1_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/nodes_used_collection_shard1_replica2/core.properties (No such file or directory)
   [junit4]   2> 759947 T2277 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 759948 T2290 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 759945 T2303 oasc.CorePropertiesLocator.writePropertiesFile ERROR Couldn't persist core properties to /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/nodes_used_collection_shard2_replica2/core.properties: java.io.FileNotFoundException: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/nodes_used_collection_shard2_replica2/core.properties (No such file or directory)
   [junit4]   2> 759949 T2290 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 759948 T2277 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 759947 T2319 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard1_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/nodes_used_collection_shard1_replica2
   [junit4]   2> 759949 T2303 oasc.CoreContainer.create Creating SolrCore 'nodes_used_collection_shard2_replica2' using instanceDir: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/nodes_used_collection_shard2_replica2
   [junit4]   2> 759951 T2303 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 759950 T2319 oasc.ZkController.createCollectionZkNode Check for collection zkNode:nodes_used_collection
   [junit4]   2> 759952 T2290 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 759952 T2303 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 759953 T2290 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 759953 T2319 oasc.ZkController.createCollectionZkNode Creating collection in ZooKeeper:nodes_used_collection
   [junit4]   2> 759953 T2303 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 759954 T2319 oasc.ZkController.getConfName Looking for collection configName
   [junit4]   2> 759954 T2277 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 759956 T2277 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 759960 T2303 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 759960 T2290 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 759960 T2303 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 759961 T2277 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 759961 T2319 oasc.ZkController.getConfName Only one config set found in zk - using it:conf1
   [junit4]   2> 759961 T2319 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection
   [junit4]   2> 759963 T2290 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/nodes_used_collection_shard2_replica1/'
   [junit4]   2> 759964 T2277 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/nodes_used_collection_shard1_replica1/'
   [junit4]   2> 759966 T2303 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 759967 T2319 oasc.ZkController.readConfigName Load collection config from:/collections/nodes_used_collection
   [junit4]   2> 759968 T2303 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/nodes_used_collection_shard2_replica2/'
   [junit4]   2> 759969 T2319 oasc.SolrResourceLoader.<init> new SolrResourceLoader for directory: '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/nodes_used_collection_shard1_replica2/'
   [junit4]   2> 760004 T2290 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 760019 T2277 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 760022 T2319 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 760026 T2303 oasc.SolrConfig.<init> Using Lucene MatchVersion: LUCENE_45
   [junit4]   2> 760042 T2290 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 760044 T2290 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 760051 T2290 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica1] Schema name=test
   [junit4]   2> 760062 T2277 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 760064 T2277 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 760071 T2277 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica1] Schema name=test
   [junit4]   2> 760080 T2319 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 760082 T2319 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 760083 T2303 oasc.SolrConfig.<init> Loaded SolrConfig: solrconfig.xml
   [junit4]   2> 760085 T2303 oass.IndexSchema.readSchema Reading Solr Schema from schema.xml
   [junit4]   2> 760091 T2319 oass.IndexSchema.readSchema [nodes_used_collection_shard1_replica2] Schema name=test
   [junit4]   2> 760101 T2303 oass.IndexSchema.readSchema [nodes_used_collection_shard2_replica2] Schema name=test
   [junit4]   2> 760504 T2277 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760504 T2277 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760505 T2277 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 760514 T2277 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 760519 T2277 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 760543 T2277 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760548 T2277 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760554 T2277 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760556 T2277 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760556 T2277 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760557 T2277 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760559 T2277 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760559 T2277 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760559 T2290 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760560 T2277 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 760560 T2290 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760560 T2277 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/nodes_used_collection_shard1_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313//nodes_used_collection_shard1_replica1/data/
   [junit4]   2> 760561 T2290 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 760561 T2277 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5882
   [junit4]   2> 760562 T2277 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313//nodes_used_collection_shard1_replica1/data
   [junit4]   2> 760563 T2277 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313//nodes_used_collection_shard1_replica1/data/index/
   [junit4]   2> 760563 T2277 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313/nodes_used_collection_shard1_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 760564 T2277 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty1-1375500974313//nodes_used_collection_shard1_replica1/data/index
   [junit4]   2> 760566 T2277 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@1beb8c8 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1766392),segFN=segments_1,generation=1}
   [junit4]   2> 760566 T2277 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 760567 T2290 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 760570 T2290 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 760571 T2277 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 760571 T2277 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 760572 T2277 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 760573 T2277 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 760573 T2277 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 760574 T2277 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 760574 T2277 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 760575 T2277 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 760575 T2277 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 760578 T2277 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 760581 T2277 oass.SolrIndexSearcher.<init> Opening Searcher@b3842 main
   [junit4]   2> 760582 T2277 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 760582 T2277 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 760586 T2290 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760586 T2329 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica1] Registered new searcher Searcher@b3842 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 760589 T2277 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica1
   [junit4]   2> 760589 T2277 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica1 address:http://127.0.0.1:45099 collection:nodes_used_collection shard:shard1
   [junit4]   2> 760590 T2277 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 760591 T2290 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760594 T2290 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760596 T2277 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard1/election
   [junit4]   2> 760597 T2290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760597 T2290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760597 T2290 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760600 T2290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760600 T2290 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760601 T2290 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 760601 T2290 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica1] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/nodes_used_collection_shard2_replica1/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518//nodes_used_collection_shard2_replica1/data/
   [junit4]   2> 760601 T2290 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5882
   [junit4]   2> 760602 T2290 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518//nodes_used_collection_shard2_replica1/data
   [junit4]   2> 760603 T2290 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518//nodes_used_collection_shard2_replica1/data/index/
   [junit4]   2> 760603 T2290 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica1] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518/nodes_used_collection_shard2_replica1/data/index' doesn't exist. Creating new index...
   [junit4]   2> 760604 T2290 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty2-1375500975518//nodes_used_collection_shard2_replica1/data/index
   [junit4]   2> 760606 T2290 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@78f638 lockFactory=org.apache.lucene.store.NativeFSLockFactory@2cf608),segFN=segments_1,generation=1}
   [junit4]   2> 760606 T2290 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 760610 T2290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 760611 T2290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 760612 T2277 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard1
   [junit4]   2> 760612 T2290 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 760613 T2290 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 760613 T2290 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 760614 T2290 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 760614 T2290 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 760615 T2290 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 760615 T2277 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard1: total=2 found=1 timeoutin=180000
   [junit4]   2> 760615 T2290 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 760618 T2290 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 760619 T2290 oass.SolrIndexSearcher.<init> Opening Searcher@1fcd95 main
   [junit4]   2> 760620 T2290 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 760620 T2290 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 760624 T2330 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica1] Registered new searcher Searcher@1fcd95 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 760626 T2290 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica1
   [junit4]   2> 760627 T2290 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica1 address:http://127.0.0.1:54076 collection:nodes_used_collection shard:shard2
   [junit4]   2> 760627 T2290 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 760632 T2290 oascc.SolrZkClient.makePath makePath: /collections/nodes_used_collection/leader_elect/shard2/election
   [junit4]   2> 760632 T2319 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760633 T2319 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760634 T2319 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 760644 T2319 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 760647 T2303 oass.ByteField.init WARN ByteField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760647 T2290 oasc.ShardLeaderElectionContext.runLeaderProcess Running the leader process for shard shard2
   [junit4]   2> 760647 T2319 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 760647 T2303 oass.ShortField.init WARN ShortField is deprecated and will be removed in 5.0. You should use TrieIntField instead.
   [junit4]   2> 760649 T2303 oass.OpenExchangeRatesOrgProvider.init Initialized with rates=open-exchange-rates.json, refreshInterval=1440.
   [junit4]   2> 760652 T2290 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Waiting until we see more replicas up for shard shard2: total=2 found=1 timeoutin=179999
   [junit4]   2> 760654 T2303 oass.IndexSchema.readSchema default search field in schema is text
   [junit4]   2> 760657 T2303 oass.IndexSchema.readSchema unique key field: id
   [junit4]   2> 760667 T2319 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760673 T2319 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760677 T2303 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760677 T2319 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760679 T2319 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760680 T2319 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760680 T2319 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760681 T2303 oass.FileExchangeRateProvider.reload Reloading exchange rates from file currency.xml
   [junit4]   2> 760681 T2319 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760681 T2319 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760682 T2319 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 760682 T2319 oasc.SolrCore.<init> [nodes_used_collection_shard1_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/nodes_used_collection_shard1_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//nodes_used_collection_shard1_replica2/data/
   [junit4]   2> 760682 T2319 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5882
   [junit4]   2> 760683 T2319 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//nodes_used_collection_shard1_replica2/data
   [junit4]   2> 760684 T2319 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//nodes_used_collection_shard1_replica2/data/index/
   [junit4]   2> 760684 T2319 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard1_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913/nodes_used_collection_shard1_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 760684 T2319 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//nodes_used_collection_shard1_replica2/data/index
   [junit4]   2> 760684 T2303 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760686 T2319 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@e83791 lockFactory=org.apache.lucene.store.NativeFSLockFactory@11df6a6),segFN=segments_1,generation=1}
   [junit4]   2> 760686 T2319 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 760686 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760687 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760687 T2303 oass.OpenExchangeRatesOrgProvider.reload Reloading exchange rates from open-exchange-rates.json
   [junit4]   2> 760688 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Unknown key IMPORTANT NOTE
   [junit4]   2> 760689 T2303 oass.OpenExchangeRatesOrgProvider$OpenExchangeRates.<init> WARN Expected key, got STRING
   [junit4]   2> 760689 T2319 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 760689 T2303 oasc.SolrCore.initDirectoryFactory org.apache.solr.core.MockDirectoryFactory
   [junit4]   2> 760689 T2319 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 760690 T2303 oasc.SolrCore.<init> [nodes_used_collection_shard2_replica2] Opening new SolrCore at /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/nodes_used_collection_shard2_replica2/, dataDir=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724//nodes_used_collection_shard2_replica2/data/
   [junit4]   2> 760690 T2319 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 760690 T2303 oasc.JmxMonitoredMap.<init> JMX monitoring is enabled. Adding Solr mbeans to JMX Server: com.sun.jmx.mbeanserver.JmxMBeanServer@1aa5882
   [junit4]   2> 760691 T2319 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 760691 T2319 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 760692 T2303 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724//nodes_used_collection_shard2_replica2/data
   [junit4]   2> 760692 T2319 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 760692 T2303 oasc.SolrCore.getNewIndexDir New index directory detected: old=null new=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724//nodes_used_collection_shard2_replica2/data/index/
   [junit4]   2> 760692 T2319 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 760693 T2303 oasc.SolrCore.initIndex WARN [nodes_used_collection_shard2_replica2] Solr index directory '/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724/nodes_used_collection_shard2_replica2/data/index' doesn't exist. Creating new index...
   [junit4]   2> 760693 T2319 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 760694 T2303 oasc.CachingDirectoryFactory.get return new directory for /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty3-1375500976724//nodes_used_collection_shard2_replica2/data/index
   [junit4]   2> 760694 T2319 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 760697 T2303 oasc.SolrDeletionPolicy.onCommit SolrDeletionPolicy.onCommit: commits: num=1
   [junit4]   2> commit{dir=MockDirWrapper(org.apache.lucene.store.RAMDirectory@bc1165 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1c0280e),segFN=segments_1,generation=1}
   [junit4]   2> 760697 T2303 oasc.SolrDeletionPolicy.updateCommits newest commit generation = 1
   [junit4]   2> 760697 T2319 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 760700 T2319 oass.SolrIndexSearcher.<init> Opening Searcher@1f6cae6 main
   [junit4]   2> 760701 T2319 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 760701 T2319 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 760702 T2303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "dedupe"
   [junit4]   2> 760702 T2303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "stored_sig"
   [junit4]   2> 760703 T2303 oasup.UpdateRequestProcessorChain.init inserting DistributedUpdateProcessorFactory into updateRequestProcessorChain "distrib-dup-test-chain-implicit"
   [junit4]   2> 760704 T2303 oasc.RequestHandlers.initHandlersFromConfig created standard: solr.StandardRequestHandler
   [junit4]   2> 760705 T2303 oasc.RequestHandlers.initHandlersFromConfig created /get: solr.RealTimeGetHandler
   [junit4]   2> 760705 T2331 oasc.SolrCore.registerSearcher [nodes_used_collection_shard1_replica2] Registered new searcher Searcher@1f6cae6 main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 760705 T2303 oasc.RequestHandlers.initHandlersFromConfig adding lazy requestHandler: solr.ReplicationHandler
   [junit4]   2> 760706 T2303 oasc.RequestHandlers.initHandlersFromConfig created /replication: solr.ReplicationHandler
   [junit4]   2> 760706 T2303 oasc.RequestHandlers.initHandlersFromConfig created /update: solr.UpdateRequestHandler
   [junit4]   2> 760706 T2319 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard1_replica2
   [junit4]   2> 760707 T2303 oasc.RequestHandlers.initHandlersFromConfig created /admin/: org.apache.solr.handler.admin.AdminHandlers
   [junit4]   2> 760707 T2319 oasc.ZkController.register Register replica - core:nodes_used_collection_shard1_replica2 address:http://127.0.0.1:37275 collection:nodes_used_collection shard:shard1
   [junit4]   2> 760708 T2319 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 760710 T2303 oashl.XMLLoader.init xsltCacheLifetimeSeconds=60
   [junit4]   2> 760713 T2303 oass.SolrIndexSearcher.<init> Opening Searcher@17253aa main
   [junit4]   2> 760714 T2303 oasu.CommitTracker.<init> Hard AutoCommit: disabled
   [junit4]   2> 760715 T2303 oasu.CommitTracker.<init> Soft AutoCommit: disabled
   [junit4]   2> 760719 T2332 oasc.SolrCore.registerSearcher [nodes_used_collection_shard2_replica2] Registered new searcher Searcher@17253aa main{StandardDirectoryReader(segments_1:1)}
   [junit4]   2> 760721 T2303 oasc.CoreContainer.registerCore registering core: nodes_used_collection_shard2_replica2
   [junit4]   2> 760721 T2303 oasc.ZkController.register Register replica - core:nodes_used_collection_shard2_replica2 address:http://127.0.0.1:38163 collection:nodes_used_collection shard:shard2
   [junit4]   2> 760721 T2303 oascsi.HttpClientUtil.createClient Creating new http client, config:maxConnections=10000&maxConnectionsPerHost=20&connTimeout=15000&socketTimeout=30000&retry=false
   [junit4]   2> 760957 T2264 oascc.ZkStateReader.updateClusterState Updating cloud state from ZooKeeper...
   [junit4]   2> 760967 T2295 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760967 T2281 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760967 T2263 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760968 T2268 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760968 T2309 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 760967 T2323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json, has occurred - updating... (live nodes size: 5)
   [junit4]   2> 761117 T2277 oasc.ShardLeaderElectionContext.waitForReplicasToComeUp Enough replicas found to continue.
   [junit4]   2> 761118 T2277 oasc.ShardLeaderElectionContext.runLeaderProcess I may be the new leader - try and sync
   [junit4]   2> 761119 T2277 oasc.SyncStrategy.sync Sync replicas to http://127.0.0.1:45099/nodes_used_collection_shard1_replica1/
   [junit4]   2> 761119 T2277 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:45099 START replicas=[http://127.0.0.1:37275/nodes_used_collection_shard1_replica2/] nUpdates=100
   [junit4]   2> 761120 T2277 oasu.PeerSync.sync PeerSync: core=nodes_used_collection_shard1_replica1 url=http://127.0.0.1:45099 DONE.  We have no versions.  sync failed.
   [junit4]   2> 761121 T2277 oasc.SyncStrategy.syncReplicas Leader's attempt to sync with shard failed, moving to the next candidate
   [junit4]   2> 761121 T2277 oasc.ShardLeaderElectionContext.runLeaderProcess We failed sync, but we have no versions - we can't sync in that case - we were active before, so become leader anyway
   [junit4]   2> 761122 T2277 oasc.ShardLeaderElectionContext.runLeaderProcess I am the new leader: http://127.0.0.1:45099/nodes_used_collection_shard1_

[...truncated too long message...]

0,rollbacks=0,expungeDeletes=0,docsPending=0,adds=0,deletesById=0,deletesByQuery=0,errors=0,cumulative_adds=0,cumulative_deletesById=0,cumulative_deletesByQuery=0,cumulative_errors=0}
   [junit4]   2> 884228 T2241 oasu.SolrCoreState.decrefSolrCoreState Closing SolrCoreState
   [junit4]   2> 884228 T2241 oasu.DefaultSolrCoreState.closeIndexWriter SolrCoreState ref count has reached 0 - closing IndexWriter
   [junit4]   2> 884229 T2241 oasu.DefaultSolrCoreState.closeIndexWriter closing IndexWriter with IndexWriterCloser
   [junit4]   2> 884229 T2241 oasc.SolrCore.closeSearcher [new_implicit_coll_1_b_replica2] Closing main searcher on request.
   [junit4]   2> 884229 T2241 oasc.CachingDirectoryFactory.close Closing MockDirectoryFactory - 2 directories currently being tracked
   [junit4]   2> 884230 T2241 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//new_implicit_coll_1_b_replica2/data/index [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//new_implicit_coll_1_b_replica2/data/index;done=false>>]
   [junit4]   2> 884230 T2241 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//new_implicit_coll_1_b_replica2/data/index
   [junit4]   2> 884230 T2241 oasc.CachingDirectoryFactory.closeCacheValue looking to close /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//new_implicit_coll_1_b_replica2/data [CachedDir<<refCount=0;path=/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//new_implicit_coll_1_b_replica2/data;done=false>>]
   [junit4]   2> 884231 T2241 oasc.CachingDirectoryFactory.close Closing directory: /mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build/solr-core/test/J0/./org.apache.solr.cloud.CollectionsAPIDistributedZkTest-jetty4-1375500977913//new_implicit_coll_1_b_replica2/data
   [junit4]   2> 884232 T2758 oasc.Overseer$ClusterStateUpdater.amILeader According to ZK I (id=90144831822037004-127.0.0.1:37275_-n_0000000004) am no longer a leader.
   [junit4]   2> 884256 T2241 oejsh.ContextHandler.doStop stopped o.e.j.s.ServletContextHandler{/,null}
   [junit4]   2> 884309 T2241 oas.SolrTestCaseJ4.tearDown ###Ending testDistribSearch
   [junit4]   2> 884311 T2241 oasc.ZkTestServer.send4LetterWord connecting to 127.0.0.1:41165 41165
   [junit4]   1> client port:0.0.0.0/0.0.0.0:0
   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=CollectionsAPIDistributedZkTest -Dtests.method=testDistribSearch -Dtests.seed=28BC2E40482BEFCE -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=ar_LY -Dtests.timezone=Africa/Dakar -Dtests.file.encoding=UTF-8
   [junit4] ERROR    133s J0 | CollectionsAPIDistributedZkTest.testDistribSearch <<<
   [junit4]    > Throwable #1: org.apache.solr.client.solrj.SolrServerException: Server refused connection at: http://127.0.0.1:54076
   [junit4]    > at __randomizedtesting.SeedInfo.seed([28BC2E40482BEFCE:A95AA0583F748FF2]:0)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:428)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:180)
   [junit4]    > at org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1535)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testCustomCollectionsAPI(CollectionsAPIDistributedZkTest.java:677)
   [junit4]    > at org.apache.solr.cloud.CollectionsAPIDistributedZkTest.doTest(CollectionsAPIDistributedZkTest.java:159)
   [junit4]    > at org.apache.solr.BaseDistributedSearchTestCase.testDistribSearch(BaseDistributedSearchTestCase.java:835)
   [junit4]    > at java.lang.Thread.run(Thread.java:662)
   [junit4]    > Caused by: org.apache.http.conn.HttpHostConnectException: Connection to http://127.0.0.1:54076 refused
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
   [junit4]    > at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:645)
   [junit4]    > at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:480)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
   [junit4]    > at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
   [junit4]    > at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:365)
   [junit4]    > ... 45 more
   [junit4]    > Caused by: java.net.ConnectException: Connection refused
   [junit4]    > at java.net.PlainSocketImpl.socketConnect(Native Method)
   [junit4]    > at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
   [junit4]    > at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
   [junit4]    > at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
   [junit4]    > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
   [junit4]    > at java.net.Socket.connect(Socket.java:529)
   [junit4]    > at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:127)
   [junit4]    > at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
   [junit4]    > ... 52 more
   [junit4]   2> 884333 T2241 oas.SolrTestCaseJ4.deleteCore ###deleteCore
   [junit4]   2> 132885 T2240 ccr.ThreadLeakControl.checkThreadLeaks WARNING Will linger awaiting termination of 2 leaked thread(s).
   [junit4]   2> 885534 T2323 oascc.ZkStateReader$3.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> 885534 T2323 oascc.ZkStateReader$2.process A cluster state change: WatchedEvent state:SyncConnected type:NodeChildrenChanged path:/live_nodes, has occurred - updating... (live nodes size: 1)
   [junit4]   2> 885534 T2323 oascc.ZkStateReader$2.process WARN ZooKeeper watch triggered, but Solr cannot talk to ZK
   [junit4]   2> NOTE: test params are: codec=Lucene40, sim=DefaultSimilarity, locale=ar_LY, timezone=Africa/Dakar
   [junit4]   2> NOTE: Linux 3.2.0-51-generic i386/Sun Microsystems Inc. 1.6.0_45 (32-bit)/cpus=8,threads=1,free=183341648,total=351731712
   [junit4]   2> NOTE: All tests run in this JVM: [UpdateRequestProcessorFactoryTest, TestXIncludeConfig, TestPerFieldSimilarity, TestSearchPerf, TestFuzzyAnalyzedSuggestions, TestCoreContainer, SuggesterTest, SolrCoreTest, PrimUtilsTest, SearchHandlerTest, CSVRequestHandlerTest, MBeansHandlerTest, TestTrie, QueryParsingTest, LeaderElectionTest, TestFastLRUCache, TestJoin, SolrInfoMBeanTest, TestConfig, TestRemoteStreaming, TestStressVersions, SliceStateTest, BasicDistributedZkTest, TestSweetSpotSimilarityFactory, BadComponentTest, DocumentBuilderTest, TestFastWriter, TestFieldTypeResource, SyncSliceTest, DistributedSpellCheckComponentTest, SystemInfoHandlerTest, TestSolrXml, MinimalSchemaTest, TestSolrQueryParser, ShardRoutingTest, ScriptEngineTest, TestDynamicFieldCollectionResource, SolrIndexSplitterTest, TestDefaultSearchFieldResource, BlockDirectoryTest, QueryEqualityTest, LegacyHTMLStripCharFilterTest, UUIDFieldTest, TestSolrIndexConfig, TestCharFilters, IndexSchemaTest, CoreMergeIndexesAdminHandlerTest, FieldMutatingUpdateProcessorTest, SoftAutoCommitTest, RegexBoostProcessorTest, TestLRUCache, ZkCLITest, TestDocSet, SliceStateUpdateTest, PeerSyncTest, TestPHPSerializedResponseWriter, TestSort, TestFieldTypeCollectionResource, ShowFileRequestHandlerTest, CoreAdminHandlerTest, TestLFUCache, DistributedQueryElevationComponentTest, TestExtendedDismaxParser, SimpleFacetsTest, MoreLikeThisHandlerTest, BinaryUpdateRequestHandlerTest, TestIBSimilarityFactory, RecoveryZkTest, TestStressRecovery, JsonLoaderTest, DistributedTermsComponentTest, DocValuesTest, ResourceLoaderTest, AutoCommitTest, DateFieldTest, CollectionsAPIDistributedZkTest]
   [junit4] Completed on J0 in 134.16s, 1 test, 1 error <<< FAILURES!

[...truncated 487 lines...]
BUILD FAILED
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:395: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:375: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/build.xml:39: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/build.xml:181: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/solr/common-build.xml:449: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:1233: The following error occurred while executing this line:
/mnt/ssd/jenkins/workspace/Lucene-Solr-4.x-Linux/lucene/common-build.xml:876: There were test failures: 321 suites, 1352 tests, 1 error, 34 ignored (6 assumptions)

Total time: 40 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Description set: Java: 32bit/jdk1.6.0_45 -server -XX:+UseParallelGC
Archiving artifacts
Recording test results
Email was triggered for: Failure
Sending email for trigger: Failure




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]