SolrCloud : node recovery fails with "No registered leader was found"

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

SolrCloud : node recovery fails with "No registered leader was found"

Zisis T.
SolrCloud configuration contains a single shard and 2 Solr servers, therefore one acts as a leader and one as a replica.

Through a series of events(*) I've ended up with one Solr server being in "Active" status and the leader of the shard while the other one in "Recovery failed" status which cannot recover no matter what. It keeps retrying every 600 sec and logs the following error

ERROR org.apache.solr.cloud.RecoveryStrategy [RecoveryThread] - Error while trying to recover. core=sample:org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: sample slice: shard1
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:531)
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:514)
        at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:345)
        at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)

Does anyone have any idea as to why the replica cannot locate the leader?
What is the proposed solution in this case?

(*) Sorry for not being able to provide more details but if it's of any help here goes
-> SolrCloud fails to start because of write.lock in index folders
-> Shutdown servers and remove write.lock files
-> Restart Zookeeper ensemble
-> Restart Solr servers
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud : node recovery fails with "No registered leader was found"

Erick Erickson
What version of Solr?

On Wed, May 21, 2014 at 2:23 AM, zzT <[hidden email]> wrote:

> SolrCloud configuration contains a single shard and 2 Solr servers, therefore
> one acts as a leader and one as a replica.
>
> Through a series of events(*) I've ended up with one Solr server being in
> "Active" status and the leader of the shard while the other one in "Recovery
> failed" status which cannot recover no matter what. It keeps retrying every
> 600 sec and logs the following error
>
> ERROR org.apache.solr.cloud.RecoveryStrategy [RecoveryThread] - Error while
> trying to recover. core=sample:org.apache.solr.common.SolrException: No
> registered leader was found after waiting for 4000ms , collection: sample
> slice: shard1
>         at
> org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:531)
>         at
> org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:514)
>         at
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:345)
>         at
> org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
>
> Does anyone have any idea as to why the replica cannot locate the leader?
> What is the proposed solution in this case?
>
> (*) Sorry for not being able to provide more details but if it's of any help
> here goes
> -> SolrCloud fails to start because of write.lock in index folders
> -> Shutdown servers and remove write.lock files
> -> Restart Zookeeper ensemble
> -> Restart Solr servers
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-node-recovery-fails-with-No-registered-leader-was-found-tp4137331.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud : node recovery fails with "No registered leader was found"

Zisis T.
I'm using Solr 4.7.2.

A few things I've missed follow. Before reaching the "one leader-one failed to recover" state, the situation was no leader for the shard and both nodes in "recovery failed" mode. A bit of tinkering to clusterstate.json "forced" the one to be the leader but that didn't change a thing. The error message "No registered leader was found after waiting" was the same before and after tinkering.

The other thing I found out is that there was a period when ZK ensemble was down for a while. In Solr log and upon startup I found ithe following
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /collections/sample/leaders/shard1

It seems though that the Solr servers could not communicate with ZK once it was functional again.
Restarting the servers once more fixed the issue and the failing server managed to "recover" the index successfully.



Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud : node recovery fails with "No registered leader was found"

heaven
Seeing the same thing after a crash of one ZK node (from 5):
{code}
org.apache.solr.common.SolrException: No registered leader was found after waiting for 4000ms , collection: crm-prod slice: shard1
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:545)
        at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:528)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.setupRequest(DistributedUpdateProcessor.java:250)
        at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:982)
        at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
        at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:349)
        at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:278)
        at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
        at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
        at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
        at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
        at org.apache.solr.core.SolrCore.execute(SolrCore.java:1952)
        at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:774)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
        at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
        at org.eclipse.jetty.server.Server.handle(Server.java:368)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
        at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
        at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
        at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
        at java.lang.Thread.run(Thread.java:744)
{code}
Reply | Threaded
Open this post in threaded view
|

Re: SolrCloud : node recovery fails with "No registered leader was found"

yann180
Hi guys,

just wondering if any solution was found for this?

I have a similar problem - Solr 4.7.2, 2-server cloud, single replicated shard.

At random times one of the server dies with a the same message as in the title of this thread.

I was hoping there might be a solution? (upgrading Solr is not practical for me because of the JDK 1.7 requirement).

Thanks

Yann