Leader is stuck on offline node

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Leader is stuck on offline node

Bernd Fehling
Something strange happened,
in my Solr 6.6.5 cloud (1 collection, 5 shards, 3 replica) the
leader is stuck on offline node for shard3.

I already tried setting property preferredLeader to true on the
active core_node5 and called REBALANCELEADERS but nothing happened.
In the response of REBALANCELEADERS was nothing about shard3.

It feels like it doesn't know anything about core_node5.

Any idea how to fix this?

<lst name="shard3">
   <str name="range">e6660000-1998ffff</str>
   <str name="state">active</str>
   <lst name="replicas">
     <lst name="core_node5">
       <str name="core">base1_shard3_replica1</str>
       <str name="base_url">http://server05.myip.com:8983/solr</str>
       <str name="node_name">server05.myip.com:8983_solr</str>
       <str name="state">active</str>
       <str name="property.preferredleader">true</str>
     </lst>
     <lst name="core_node7">
       <str name="core">base1_shard3_replica2</str>
       <str name="base_url">http://server02.myip.com:8983/solr</str>
       <str name="node_name">server02.myip.com:8983_solr</str>
       <str name="state">down</str>
       <str name="leader">true</str>
     </lst>
     <lst name="core_node11">
       <str name="core">base1_shard3_replica3</str>
       <str name="base_url">http://server03.myip.com:8983/solr</str>
       <str name="node_name">server03.myip.com:8983_solr</str>
       <str name="state">down</str>
     </lst>
   </lst>
</lst>

Regards,
Bernd

Reply | Threaded
Open this post in threaded view
|

Re: Leader is stuck on offline node

Erick Erickson
REBALANCELEADERS and related do not force a leader to be elected,
rather it tries to respect the preferredLeader property if it's been
set in the replica's properties. Really that's only intended for
situations where there are many 10s of leaders (perhaps hundreds) that
happen to be on the same host.

If you're reasonably sure base1_shard3_replica1 is healthy, try FORCELEADER.

Best,
Erick

On Thu, Aug 9, 2018 at 4:56 AM, Bernd Fehling
<[hidden email]> wrote:

> Something strange happened,
> in my Solr 6.6.5 cloud (1 collection, 5 shards, 3 replica) the
> leader is stuck on offline node for shard3.
>
> I already tried setting property preferredLeader to true on the
> active core_node5 and called REBALANCELEADERS but nothing happened.
> In the response of REBALANCELEADERS was nothing about shard3.
>
> It feels like it doesn't know anything about core_node5.
>
> Any idea how to fix this?
>
> <lst name="shard3">
>   <str name="range">e6660000-1998ffff</str>
>   <str name="state">active</str>
>   <lst name="replicas">
>     <lst name="core_node5">
>       <str name="core">base1_shard3_replica1</str>
>       <str name="base_url">http://server05.myip.com:8983/solr</str>
>       <str name="node_name">server05.myip.com:8983_solr</str>
>       <str name="state">active</str>
>       <str name="property.preferredleader">true</str>
>     </lst>
>     <lst name="core_node7">
>       <str name="core">base1_shard3_replica2</str>
>       <str name="base_url">http://server02.myip.com:8983/solr</str>
>       <str name="node_name">server02.myip.com:8983_solr</str>
>       <str name="state">down</str>
>       <str name="leader">true</str>
>     </lst>
>     <lst name="core_node11">
>       <str name="core">base1_shard3_replica3</str>
>       <str name="base_url">http://server03.myip.com:8983/solr</str>
>       <str name="node_name">server03.myip.com:8983_solr</str>
>       <str name="state">down</str>
>     </lst>
>   </lst>
> </lst>
>
> Regards,
> Bernd
>