solr4.1 No live SolrServers available to handle this request

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

solr4.1 No live SolrServers available to handle this request

sling
hi,all. I am new to Solr.
when i query solrcloud4.1 with solrj, the client throws exceptions as follows.
there are 2 shards in my solrcloud.  
each shard is on a server with 4cpu/3G RAM, and jvm has 2G ram.
when the query requests get more and more, the exception occers.
     [java] org.apache.solr.client.solrj.SolrServerException: No live SolrServers available to handle this request
     [java]     at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:486)
     [java]     at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
     [java]     at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
     [java]     at com.netease.index.service.impl.SearcherServiceImpl.search(Unknown Source)
     [java]     at com.netease.index.util.ConSearcher.run(Unknown Source)
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java]     at java.lang.Thread.run(Thread.java:662)
     [java] Caused by: org.apache.solr.client.solrj.SolrServerException: IOException occured when talking to server at: http://cms.test.com/solr/doc
     [java]     at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:416)
     [java]     at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
     [java]     at org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:439)
     [java]     ... 7 more
     [java] Caused by: org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
     [java]     at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:416)
     [java]     at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:299)
     [java]     at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:242)
     [java]     at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:455)
     [java]     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
     [java]     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
     [java]     at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
     [java]     at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
     [java]     ... 9 more

ps: lbhttpsolrserver seems to allocate task imbalance...some node get a much heavy load, while others may be not.   i use nginx so that task could be more controllable.  is this right?


please help me out, Thank you in advance. ^_^




Reply | Threaded
Open this post in threaded view
|

Re: solr4.1 No live SolrServers available to handle this request

Joel Bernstein
Check the Solr logs for Zookeeper disconnects. It could be that as load is
increasing Solr is not able to respond to the Zookeeper pings which would
bring the nodes offline. If you see Zookeeper disconnects then you can
increase the zkClientTimeout set in solr.xml. But be aware that zk
disconnects can also be a sign that your servers are overload and/or under
resourced. Memory starvation and stop the world GC can often be the cause
of zk disconnects.


On Mon, Apr 1, 2013 at 6:18 AM, sling <[hidden email]> wrote:

> hi,all. I am new to Solr.
> when i query solrcloud4.1 with solrj, the client throws exceptions as
> follows.
> there are 2 shards in my solrcloud.
> each shard is on a server with 4cpu/3G RAM, and jvm has 2G ram.
> when the query requests get more and more, the exception occers.
>      [java] org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers available to handle this request
>      [java]     at
>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:486)
>      [java]     at
>
> org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
>      [java]     at
> org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
>      [java]     at
> com.netease.index.service.impl.SearcherServiceImpl.search(Unknown Source)
>      [java]     at com.netease.index.util.ConSearcher.run(Unknown Source)
>      [java]     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>      [java]     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>      [java]     at java.lang.Thread.run(Thread.java:662)
>      [java] Caused by: org.apache.solr.client.solrj.SolrServerException:
> IOException occured when talking to server at:
> http://cms.test.com/solr/doc
>      [java]     at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:416)
>      [java]     at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
>      [java]     at
>
> org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:439)
>      [java]     ... 7 more
>      [java] Caused by: org.apache.http.conn.ConnectionPoolTimeoutException:
> Timeout waiting for connection from pool
>      [java]     at
>
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:416)
>      [java]     at
>
> org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:299)
>      [java]     at
>
> org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:242)
>      [java]     at
>
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:455)
>      [java]     at
>
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
>      [java]     at
>
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
>      [java]     at
>
> org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
>      [java]     at
>
> org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:353)
>      [java]     ... 9 more
>
> ps: lbhttpsolrserver seems to allocate task imbalance...some node get a
> much
> heavy load, while others may be not.   i use nginx so that task could be
> more controllable.  is this right?
>
>
> please help me out, Thank you in advance. ^_^
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/solr4-1-No-live-SolrServers-available-to-handle-this-request-tp4052862.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



--
Joel Bernstein
Professional Services LucidWorks
Reply | Threaded
Open this post in threaded view
|

Re: solr4.1 No live SolrServers available to handle this request

sling
thx for your reply.
my solr.xml is like this:
<solr persistent="true">
  <cores adminPath="/admin/cores" defaultCoreName="doc" host="${host:cms1.test.com}" hostPort="${jetty.port:9090}"
                hostContext="${hostContext:}" zkClientTimeout="${zkClientTimeout:30000}" leaderVoteWait="${leaderVoteWait:20000}">
        <core name="doc" instanceDir="doc/" loadOnStartup="true" transient="false" collection="docCollection" />
  </cores>
</solr>

i have change the zkclienttimeout from 15s to 30s,  but this exception still shows.........
and the load on solrcloud servers   are not too heavy, they are 1.4 1.5 1.

and these disconnects appear in solrj logs, while the solrcloud is fine.