REBALANCELEADERS is not reliable

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

REBALANCELEADERS is not reliable

Bernd Fehling
Hi list,

unfortunately REBALANCELEADERS is not reliable and the leader
election has unpredictable results with SolrCloud 6.6.5 and
Zookeeper 3.4.10.
Seen with 5 shards / 3 replicas.

- CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
- setting with ADDREPLICAPROP the property preferredLeader to other replicas
- calling REBALANCELEADERS
- some leaders have changed, some not.

I then tried:
- removing all preferredLeader properties from replicas which succeeded.
- trying again REBALANCELEADERS for the rest. No success.
- Shutting down nodes to force the leader to a specific replica left running.
   No success.
- calling REBALANCELEADERS responds that the replica is inactive!!!
- calling CLUSTERSTATUS reports that the replica is active!!!

Also, the replica which don't want to become leader is not in the list
of collections->[collection_name]->leader_elect->shard1..x->election

Where is CLUSTERSTATUS getting it's state info from?

Has anyone else problems with REBALANCELEADERS?

I noticed that the Reference Guide writes "preferredLeader" (with capital "L")
but the JAVA code has "preferredleader".

Regards, Bernd
Reply | Threaded
Open this post in threaded view
|

RE: REBALANCELEADERS is not reliable

Vadim Ivanov
Hi, Bernd
I have tried REBALANCELEADERS with Solr 6.3 and 7.5
I had very similar results and notion that it's not reliable :(
--
Br, Vadim


> -----Original Message-----
> From: Bernd Fehling [mailto:[hidden email]]
> Sent: Tuesday, November 27, 2018 5:13 PM
> To: [hidden email]
> Subject: REBALANCELEADERS is not reliable
>
> Hi list,
>
> unfortunately REBALANCELEADERS is not reliable and the leader
> election has unpredictable results with SolrCloud 6.6.5 and
> Zookeeper 3.4.10.
> Seen with 5 shards / 3 replicas.
>
> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> - setting with ADDREPLICAPROP the property preferredLeader to other replicas
> - calling REBALANCELEADERS
> - some leaders have changed, some not.
>
> I then tried:
> - removing all preferredLeader properties from replicas which succeeded.
> - trying again REBALANCELEADERS for the rest. No success.
> - Shutting down nodes to force the leader to a specific replica left running.
>    No success.
> - calling REBALANCELEADERS responds that the replica is inactive!!!
> - calling CLUSTERSTATUS reports that the replica is active!!!
>
> Also, the replica which don't want to become leader is not in the list
> of collections->[collection_name]->leader_elect->shard1..x->election
>
> Where is CLUSTERSTATUS getting it's state info from?
>
> Has anyone else problems with REBALANCELEADERS?
>
> I noticed that the Reference Guide writes "preferredLeader" (with capital "L")
> but the JAVA code has "preferredleader".
>
> Regards, Bernd

Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Bernd Fehling
In reply to this post by Bernd Fehling
Hi Vadim,

thanks for confirming.
So it seems to be a general problem with Solr 6.x, 7.x and might
be still there in the most recent versions.

But where to start to debug this problem, is it something not
correctly stored in zookeeper or is overseer the problem?

I was also reading something about a "leader queue" where possible
leaders have to be requeued or something similar.

May be I should try to get a situation where a "locked" core
is on the overseer and then connect the debugger to it and step
through it.
Peeking and poking around, like old Commodore 64 days :-)

Regards, Bernd


Am 27.11.18 um 15:47 schrieb Vadim Ivanov:

> Hi, Bernd
> I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> I had very similar results and notion that it's not reliable :(
> --
> Br, Vadim
>
>> -----Original Message-----
>> From: Bernd Fehling [mailto:[hidden email]]
>> Sent: Tuesday, November 27, 2018 5:13 PM
>> To: [hidden email]
>> Subject: REBALANCELEADERS is not reliable
>>
>> Hi list,
>>
>> unfortunately REBALANCELEADERS is not reliable and the leader
>> election has unpredictable results with SolrCloud 6.6.5 and
>> Zookeeper 3.4.10.
>> Seen with 5 shards / 3 replicas.
>>
>> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
>> - setting with ADDREPLICAPROP the property preferredLeader to other replicas
>> - calling REBALANCELEADERS
>> - some leaders have changed, some not.
>>
>> I then tried:
>> - removing all preferredLeader properties from replicas which succeeded.
>> - trying again REBALANCELEADERS for the rest. No success.
>> - Shutting down nodes to force the leader to a specific replica left running.
>>    No success.
>> - calling REBALANCELEADERS responds that the replica is inactive!!!
>> - calling CLUSTERSTATUS reports that the replica is active!!!
>>
>> Also, the replica which don't want to become leader is not in the list
>> of collections->[collection_name]->leader_elect->shard1..x->election
>>
>> Where is CLUSTERSTATUS getting it's state info from?
>>
>> Has anyone else problems with REBALANCELEADERS?
>>
>> I noticed that the Reference Guide writes "preferredLeader" (with capital "L")
>> but the JAVA code has "preferredleader".
>>
>> Regards, Bernd
>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Aman Tandon
For me today, I deleted the leader replica of one of the two shard
collection. Then other replica of that shard was getting elected for leader.

After waiting for long tried the setting addreplicaprop preferred leader on
one of the replica then tried FORCELEADER but no luck. Then also tried
rebalance but no help. Finally have to recreate the whole collection.

Not sure what was the issue but both FORCELEADER AND REBALANCING didn't
work if there was no leader however preferred leader property was setted.

On Wed, Nov 28, 2018, 12:54 Bernd Fehling <[hidden email]
wrote:

> Hi Vadim,
>
> thanks for confirming.
> So it seems to be a general problem with Solr 6.x, 7.x and might
> be still there in the most recent versions.
>
> But where to start to debug this problem, is it something not
> correctly stored in zookeeper or is overseer the problem?
>
> I was also reading something about a "leader queue" where possible
> leaders have to be requeued or something similar.
>
> May be I should try to get a situation where a "locked" core
> is on the overseer and then connect the debugger to it and step
> through it.
> Peeking and poking around, like old Commodore 64 days :-)
>
> Regards, Bernd
>
>
> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
> > Hi, Bernd
> > I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> > I had very similar results and notion that it's not reliable :(
> > --
> > Br, Vadim
> >
> >> -----Original Message-----
> >> From: Bernd Fehling [mailto:[hidden email]]
> >> Sent: Tuesday, November 27, 2018 5:13 PM
> >> To: [hidden email]
> >> Subject: REBALANCELEADERS is not reliable
> >>
> >> Hi list,
> >>
> >> unfortunately REBALANCELEADERS is not reliable and the leader
> >> election has unpredictable results with SolrCloud 6.6.5 and
> >> Zookeeper 3.4.10.
> >> Seen with 5 shards / 3 replicas.
> >>
> >> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> >> - setting with ADDREPLICAPROP the property preferredLeader to other
> replicas
> >> - calling REBALANCELEADERS
> >> - some leaders have changed, some not.
> >>
> >> I then tried:
> >> - removing all preferredLeader properties from replicas which succeeded.
> >> - trying again REBALANCELEADERS for the rest. No success.
> >> - Shutting down nodes to force the leader to a specific replica left
> running.
> >>    No success.
> >> - calling REBALANCELEADERS responds that the replica is inactive!!!
> >> - calling CLUSTERSTATUS reports that the replica is active!!!
> >>
> >> Also, the replica which don't want to become leader is not in the list
> >> of collections->[collection_name]->leader_elect->shard1..x->election
> >>
> >> Where is CLUSTERSTATUS getting it's state info from?
> >>
> >> Has anyone else problems with REBALANCELEADERS?
> >>
> >> I noticed that the Reference Guide writes "preferredLeader" (with
> capital "L")
> >> but the JAVA code has "preferredleader".
> >>
> >> Regards, Bernd
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Aman Tandon
++ correction

On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email] wrote:

> For me today, I deleted the leader replica of one of the two shard
> collection. Then other replicas of that shard wasn't getting elected for
> leader.
>
> After waiting for long tried the setting addreplicaprop preferred leader
> on one of the replica then tried FORCELEADER but no luck. Then also tried
> rebalance but no help. Finally have to recreate the whole collection.
>
> Not sure what was the issue but both FORCELEADER AND REBALANCING didn't
> work if there was no leader however preferred leader property was setted.
>
> On Wed, Nov 28, 2018, 12:54 Bernd Fehling <[hidden email]
> wrote:
>
>> Hi Vadim,
>>
>> thanks for confirming.
>> So it seems to be a general problem with Solr 6.x, 7.x and might
>> be still there in the most recent versions.
>>
>> But where to start to debug this problem, is it something not
>> correctly stored in zookeeper or is overseer the problem?
>>
>> I was also reading something about a "leader queue" where possible
>> leaders have to be requeued or something similar.
>>
>> May be I should try to get a situation where a "locked" core
>> is on the overseer and then connect the debugger to it and step
>> through it.
>> Peeking and poking around, like old Commodore 64 days :-)
>>
>> Regards, Bernd
>>
>>
>> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
>> > Hi, Bernd
>> > I have tried REBALANCELEADERS with Solr 6.3 and 7.5
>> > I had very similar results and notion that it's not reliable :(
>> > --
>> > Br, Vadim
>> >
>> >> -----Original Message-----
>> >> From: Bernd Fehling [mailto:[hidden email]]
>> >> Sent: Tuesday, November 27, 2018 5:13 PM
>> >> To: [hidden email]
>> >> Subject: REBALANCELEADERS is not reliable
>> >>
>> >> Hi list,
>> >>
>> >> unfortunately REBALANCELEADERS is not reliable and the leader
>> >> election has unpredictable results with SolrCloud 6.6.5 and
>> >> Zookeeper 3.4.10.
>> >> Seen with 5 shards / 3 replicas.
>> >>
>> >> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
>> >> - setting with ADDREPLICAPROP the property preferredLeader to other
>> replicas
>> >> - calling REBALANCELEADERS
>> >> - some leaders have changed, some not.
>> >>
>> >> I then tried:
>> >> - removing all preferredLeader properties from replicas which
>> succeeded.
>> >> - trying again REBALANCELEADERS for the rest. No success.
>> >> - Shutting down nodes to force the leader to a specific replica left
>> running.
>> >>    No success.
>> >> - calling REBALANCELEADERS responds that the replica is inactive!!!
>> >> - calling CLUSTERSTATUS reports that the replica is active!!!
>> >>
>> >> Also, the replica which don't want to become leader is not in the list
>> >> of collections->[collection_name]->leader_elect->shard1..x->election
>> >>
>> >> Where is CLUSTERSTATUS getting it's state info from?
>> >>
>> >> Has anyone else problems with REBALANCELEADERS?
>> >>
>> >> I noticed that the Reference Guide writes "preferredLeader" (with
>> capital "L")
>> >> but the JAVA code has "preferredleader".
>> >>
>> >> Regards, Bernd
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Atita Arora
Indeed, I tried that on 7.4 & 7.5 too, indeed did not work for me as well,
even with the preferredLeader property as recommended in the documentation.
I handled it with a little hack but certainly this dint work as expected.
I can provide more details if there's a ticket.

On Thu, Nov 29, 2018 at 8:42 PM Aman Tandon <[hidden email]> wrote:

> ++ correction
>
> On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email] wrote:
>
> > For me today, I deleted the leader replica of one of the two shard
> > collection. Then other replicas of that shard wasn't getting elected for
> > leader.
> >
> > After waiting for long tried the setting addreplicaprop preferred leader
> > on one of the replica then tried FORCELEADER but no luck. Then also tried
> > rebalance but no help. Finally have to recreate the whole collection.
> >
> > Not sure what was the issue but both FORCELEADER AND REBALANCING didn't
> > work if there was no leader however preferred leader property was setted.
> >
> > On Wed, Nov 28, 2018, 12:54 Bernd Fehling <
> [hidden email]
> > wrote:
> >
> >> Hi Vadim,
> >>
> >> thanks for confirming.
> >> So it seems to be a general problem with Solr 6.x, 7.x and might
> >> be still there in the most recent versions.
> >>
> >> But where to start to debug this problem, is it something not
> >> correctly stored in zookeeper or is overseer the problem?
> >>
> >> I was also reading something about a "leader queue" where possible
> >> leaders have to be requeued or something similar.
> >>
> >> May be I should try to get a situation where a "locked" core
> >> is on the overseer and then connect the debugger to it and step
> >> through it.
> >> Peeking and poking around, like old Commodore 64 days :-)
> >>
> >> Regards, Bernd
> >>
> >>
> >> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
> >> > Hi, Bernd
> >> > I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> >> > I had very similar results and notion that it's not reliable :(
> >> > --
> >> > Br, Vadim
> >> >
> >> >> -----Original Message-----
> >> >> From: Bernd Fehling [mailto:[hidden email]]
> >> >> Sent: Tuesday, November 27, 2018 5:13 PM
> >> >> To: [hidden email]
> >> >> Subject: REBALANCELEADERS is not reliable
> >> >>
> >> >> Hi list,
> >> >>
> >> >> unfortunately REBALANCELEADERS is not reliable and the leader
> >> >> election has unpredictable results with SolrCloud 6.6.5 and
> >> >> Zookeeper 3.4.10.
> >> >> Seen with 5 shards / 3 replicas.
> >> >>
> >> >> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> >> >> - setting with ADDREPLICAPROP the property preferredLeader to other
> >> replicas
> >> >> - calling REBALANCELEADERS
> >> >> - some leaders have changed, some not.
> >> >>
> >> >> I then tried:
> >> >> - removing all preferredLeader properties from replicas which
> >> succeeded.
> >> >> - trying again REBALANCELEADERS for the rest. No success.
> >> >> - Shutting down nodes to force the leader to a specific replica left
> >> running.
> >> >>    No success.
> >> >> - calling REBALANCELEADERS responds that the replica is inactive!!!
> >> >> - calling CLUSTERSTATUS reports that the replica is active!!!
> >> >>
> >> >> Also, the replica which don't want to become leader is not in the
> list
> >> >> of collections->[collection_name]->leader_elect->shard1..x->election
> >> >>
> >> >> Where is CLUSTERSTATUS getting it's state info from?
> >> >>
> >> >> Has anyone else problems with REBALANCELEADERS?
> >> >>
> >> >> I noticed that the Reference Guide writes "preferredLeader" (with
> >> capital "L")
> >> >> but the JAVA code has "preferredleader".
> >> >>
> >> >> Regards, Bernd
> >> >
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: REBALANCELEADERS is not reliable

Vadim Ivanov
Is solr-dev forum I came across this post
http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html
May be it will shed some light?

--
Vadim

> -----Original Message-----
> From: Atita Arora [mailto:[hidden email]]
> Sent: Thursday, November 29, 2018 11:03 PM
> To: [hidden email]
> Subject: Re: REBALANCELEADERS is not reliable
>
> Indeed, I tried that on 7.4 & 7.5 too, indeed did not work for me as well,
> even with the preferredLeader property as recommended in the
> documentation.
> I handled it with a little hack but certainly this dint work as expected.
> I can provide more details if there's a ticket.
>
> On Thu, Nov 29, 2018 at 8:42 PM Aman Tandon
> <[hidden email]> wrote:
>
> > ++ correction
> >
> > On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email]
> wrote:
> >
> > > For me today, I deleted the leader replica of one of the two shard
> > > collection. Then other replicas of that shard wasn't getting elected for
> > > leader.
> > >
> > > After waiting for long tried the setting addreplicaprop preferred leader
> > > on one of the replica then tried FORCELEADER but no luck. Then also tried
> > > rebalance but no help. Finally have to recreate the whole collection.
> > >
> > > Not sure what was the issue but both FORCELEADER AND REBALANCING
> didn't
> > > work if there was no leader however preferred leader property was setted.
> > >
> > > On Wed, Nov 28, 2018, 12:54 Bernd Fehling <
> > [hidden email]
> > > wrote:
> > >
> > >> Hi Vadim,
> > >>
> > >> thanks for confirming.
> > >> So it seems to be a general problem with Solr 6.x, 7.x and might
> > >> be still there in the most recent versions.
> > >>
> > >> But where to start to debug this problem, is it something not
> > >> correctly stored in zookeeper or is overseer the problem?
> > >>
> > >> I was also reading something about a "leader queue" where possible
> > >> leaders have to be requeued or something similar.
> > >>
> > >> May be I should try to get a situation where a "locked" core
> > >> is on the overseer and then connect the debugger to it and step
> > >> through it.
> > >> Peeking and poking around, like old Commodore 64 days :-)
> > >>
> > >> Regards, Bernd
> > >>
> > >>
> > >> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
> > >> > Hi, Bernd
> > >> > I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> > >> > I had very similar results and notion that it's not reliable :(
> > >> > --
> > >> > Br, Vadim
> > >> >
> > >> >> -----Original Message-----
> > >> >> From: Bernd Fehling [mailto:[hidden email]]
> > >> >> Sent: Tuesday, November 27, 2018 5:13 PM
> > >> >> To: [hidden email]
> > >> >> Subject: REBALANCELEADERS is not reliable
> > >> >>
> > >> >> Hi list,
> > >> >>
> > >> >> unfortunately REBALANCELEADERS is not reliable and the leader
> > >> >> election has unpredictable results with SolrCloud 6.6.5 and
> > >> >> Zookeeper 3.4.10.
> > >> >> Seen with 5 shards / 3 replicas.
> > >> >>
> > >> >> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> > >> >> - setting with ADDREPLICAPROP the property preferredLeader to other
> > >> replicas
> > >> >> - calling REBALANCELEADERS
> > >> >> - some leaders have changed, some not.
> > >> >>
> > >> >> I then tried:
> > >> >> - removing all preferredLeader properties from replicas which
> > >> succeeded.
> > >> >> - trying again REBALANCELEADERS for the rest. No success.
> > >> >> - Shutting down nodes to force the leader to a specific replica left
> > >> running.
> > >> >>    No success.
> > >> >> - calling REBALANCELEADERS responds that the replica is inactive!!!
> > >> >> - calling CLUSTERSTATUS reports that the replica is active!!!
> > >> >>
> > >> >> Also, the replica which don't want to become leader is not in the
> > list
> > >> >> of collections->[collection_name]->leader_elect->shard1..x->election
> > >> >>
> > >> >> Where is CLUSTERSTATUS getting it's state info from?
> > >> >>
> > >> >> Has anyone else problems with REBALANCELEADERS?
> > >> >>
> > >> >> I noticed that the Reference Guide writes "preferredLeader" (with
> > >> capital "L")
> > >> >> but the JAVA code has "preferredleader".
> > >> >>
> > >> >> Regards, Bernd
> > >> >
> > >>
> > >
> >

Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Bernd Fehling
In reply to this post by Atita Arora
Thanks for looking this up.
It could be a hint where to jump into the code.
I wonder why they rejected a jira ticket about this problem?

Regards, Bernd

Am 06.12.18 um 16:31 schrieb Vadim Ivanov:

> Is solr-dev forum I came across this post
> http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html
> May be it will shed some light?
>
>
>> -----Original Message-----
>> From: Atita Arora [mailto:[hidden email]]
>> Sent: Thursday, November 29, 2018 11:03 PM
>> To: [hidden email]
>> Subject: Re: REBALANCELEADERS is not reliable
>>
>> Indeed, I tried that on 7.4 & 7.5 too, indeed did not work for me as well,
>> even with the preferredLeader property as recommended in the
>> documentation.
>> I handled it with a little hack but certainly this dint work as expected.
>> I can provide more details if there's a ticket.
>>
>> On Thu, Nov 29, 2018 at 8:42 PM Aman Tandon
>> <[hidden email]> wrote:
>>
>>> ++ correction
>>>
>>> On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email]
>> wrote:
>>>
>>>> For me today, I deleted the leader replica of one of the two shard
>>>> collection. Then other replicas of that shard wasn't getting elected for
>>>> leader.
>>>>
>>>> After waiting for long tried the setting addreplicaprop preferred leader
>>>> on one of the replica then tried FORCELEADER but no luck. Then also tried
>>>> rebalance but no help. Finally have to recreate the whole collection.
>>>>
>>>> Not sure what was the issue but both FORCELEADER AND REBALANCING
>> didn't
>>>> work if there was no leader however preferred leader property was setted.
>>>>
>>>> On Wed, Nov 28, 2018, 12:54 Bernd Fehling <
>>> [hidden email]
>>>> wrote:
>>>>
>>>>> Hi Vadim,
>>>>>
>>>>> thanks for confirming.
>>>>> So it seems to be a general problem with Solr 6.x, 7.x and might
>>>>> be still there in the most recent versions.
>>>>>
>>>>> But where to start to debug this problem, is it something not
>>>>> correctly stored in zookeeper or is overseer the problem?
>>>>>
>>>>> I was also reading something about a "leader queue" where possible
>>>>> leaders have to be requeued or something similar.
>>>>>
>>>>> May be I should try to get a situation where a "locked" core
>>>>> is on the overseer and then connect the debugger to it and step
>>>>> through it.
>>>>> Peeking and poking around, like old Commodore 64 days :-)
>>>>>
>>>>> Regards, Bernd
>>>>>
>>>>>
>>>>> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
>>>>>> Hi, Bernd
>>>>>> I have tried REBALANCELEADERS with Solr 6.3 and 7.5
>>>>>> I had very similar results and notion that it's not reliable :(
>>>>>> --
>>>>>> Br, Vadim
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Bernd Fehling [mailto:[hidden email]]
>>>>>>> Sent: Tuesday, November 27, 2018 5:13 PM
>>>>>>> To: [hidden email]
>>>>>>> Subject: REBALANCELEADERS is not reliable
>>>>>>>
>>>>>>> Hi list,
>>>>>>>
>>>>>>> unfortunately REBALANCELEADERS is not reliable and the leader
>>>>>>> election has unpredictable results with SolrCloud 6.6.5 and
>>>>>>> Zookeeper 3.4.10.
>>>>>>> Seen with 5 shards / 3 replicas.
>>>>>>>
>>>>>>> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
>>>>>>> - setting with ADDREPLICAPROP the property preferredLeader to other
>>>>> replicas
>>>>>>> - calling REBALANCELEADERS
>>>>>>> - some leaders have changed, some not.
>>>>>>>
>>>>>>> I then tried:
>>>>>>> - removing all preferredLeader properties from replicas which
>>>>> succeeded.
>>>>>>> - trying again REBALANCELEADERS for the rest. No success.
>>>>>>> - Shutting down nodes to force the leader to a specific replica left
>>>>> running.
>>>>>>>    No success.
>>>>>>> - calling REBALANCELEADERS responds that the replica is inactive!!!
>>>>>>> - calling CLUSTERSTATUS reports that the replica is active!!!
>>>>>>>
>>>>>>> Also, the replica which don't want to become leader is not in the
>>> list
>>>>>>> of collections->[collection_name]->leader_elect->shard1..x->election
>>>>>>>
>>>>>>> Where is CLUSTERSTATUS getting it's state info from?
>>>>>>>
>>>>>>> Has anyone else problems with REBALANCELEADERS?
>>>>>>>
>>>>>>> I noticed that the Reference Guide writes "preferredLeader" (with
>>>>> capital "L")
>>>>>>> but the JAVA code has "preferredleader".
>>>>>>>
>>>>>>> Regards, Bernd
>>>>>>
>>>>>
>>>>
>>>
>
Reply | Threaded
Open this post in threaded view
|

RE: REBALANCELEADERS is not reliable

Vadim Ivanov
I'm waiting for 7.6 or 7.5.1 and plan to apply patch from  Endika Posadas to it.
Then test again and hope it'll help
--
Vadim


> -----Original Message-----
> From: Bernd Fehling [mailto:[hidden email]]
> Sent: Friday, December 07, 2018 12:01 PM
> To: [hidden email]
> Subject: Re: REBALANCELEADERS is not reliable
>
> Thanks for looking this up.
> It could be a hint where to jump into the code.
> I wonder why they rejected a jira ticket about this problem?
>
> Regards, Bernd
>
> Am 06.12.18 um 16:31 schrieb Vadim Ivanov:
> > Is solr-dev forum I came across this post
> > http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-
> deleted-when-rebalancing-leaders-td4417040.html
> > May be it will shed some light?
> >
> >
> >> -----Original Message-----
> >> From: Atita Arora [mailto:[hidden email]]
> >> Sent: Thursday, November 29, 2018 11:03 PM
> >> To: [hidden email]
> >> Subject: Re: REBALANCELEADERS is not reliable
> >>
> >> Indeed, I tried that on 7.4 & 7.5 too, indeed did not work for me as well,
> >> even with the preferredLeader property as recommended in the
> >> documentation.
> >> I handled it with a little hack but certainly this dint work as expected.
> >> I can provide more details if there's a ticket.
> >>
> >> On Thu, Nov 29, 2018 at 8:42 PM Aman Tandon
> >> <[hidden email]> wrote:
> >>
> >>> ++ correction
> >>>
> >>> On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email]
> >> wrote:
> >>>
> >>>> For me today, I deleted the leader replica of one of the two shard
> >>>> collection. Then other replicas of that shard wasn't getting elected for
> >>>> leader.
> >>>>
> >>>> After waiting for long tried the setting addreplicaprop preferred leader
> >>>> on one of the replica then tried FORCELEADER but no luck. Then also
> tried
> >>>> rebalance but no help. Finally have to recreate the whole collection.
> >>>>
> >>>> Not sure what was the issue but both FORCELEADER AND REBALANCING
> >> didn't
> >>>> work if there was no leader however preferred leader property was
> setted.
> >>>>
> >>>> On Wed, Nov 28, 2018, 12:54 Bernd Fehling <
> >>> [hidden email]
> >>>> wrote:
> >>>>
> >>>>> Hi Vadim,
> >>>>>
> >>>>> thanks for confirming.
> >>>>> So it seems to be a general problem with Solr 6.x, 7.x and might
> >>>>> be still there in the most recent versions.
> >>>>>
> >>>>> But where to start to debug this problem, is it something not
> >>>>> correctly stored in zookeeper or is overseer the problem?
> >>>>>
> >>>>> I was also reading something about a "leader queue" where possible
> >>>>> leaders have to be requeued or something similar.
> >>>>>
> >>>>> May be I should try to get a situation where a "locked" core
> >>>>> is on the overseer and then connect the debugger to it and step
> >>>>> through it.
> >>>>> Peeking and poking around, like old Commodore 64 days :-)
> >>>>>
> >>>>> Regards, Bernd
> >>>>>
> >>>>>
> >>>>> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
> >>>>>> Hi, Bernd
> >>>>>> I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> >>>>>> I had very similar results and notion that it's not reliable :(
> >>>>>> --
> >>>>>> Br, Vadim
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Bernd Fehling [mailto:[hidden email]]
> >>>>>>> Sent: Tuesday, November 27, 2018 5:13 PM
> >>>>>>> To: [hidden email]
> >>>>>>> Subject: REBALANCELEADERS is not reliable
> >>>>>>>
> >>>>>>> Hi list,
> >>>>>>>
> >>>>>>> unfortunately REBALANCELEADERS is not reliable and the leader
> >>>>>>> election has unpredictable results with SolrCloud 6.6.5 and
> >>>>>>> Zookeeper 3.4.10.
> >>>>>>> Seen with 5 shards / 3 replicas.
> >>>>>>>
> >>>>>>> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> >>>>>>> - setting with ADDREPLICAPROP the property preferredLeader to
> other
> >>>>> replicas
> >>>>>>> - calling REBALANCELEADERS
> >>>>>>> - some leaders have changed, some not.
> >>>>>>>
> >>>>>>> I then tried:
> >>>>>>> - removing all preferredLeader properties from replicas which
> >>>>> succeeded.
> >>>>>>> - trying again REBALANCELEADERS for the rest. No success.
> >>>>>>> - Shutting down nodes to force the leader to a specific replica left
> >>>>> running.
> >>>>>>>    No success.
> >>>>>>> - calling REBALANCELEADERS responds that the replica is inactive!!!
> >>>>>>> - calling CLUSTERSTATUS reports that the replica is active!!!
> >>>>>>>
> >>>>>>> Also, the replica which don't want to become leader is not in the
> >>> list
> >>>>>>> of collections->[collection_name]->leader_elect->shard1..x->election
> >>>>>>>
> >>>>>>> Where is CLUSTERSTATUS getting it's state info from?
> >>>>>>>
> >>>>>>> Has anyone else problems with REBALANCELEADERS?
> >>>>>>>
> >>>>>>> I noticed that the Reference Guide writes "preferredLeader" (with
> >>>>> capital "L")
> >>>>>>> but the JAVA code has "preferredleader".
> >>>>>>>
> >>>>>>> Regards, Bernd
> >>>>>>
> >>>>>
> >>>>
> >>>
> >

Reply | Threaded
Open this post in threaded view
|

RE: REBALANCELEADERS is not reliable

Vadim Ivanov
Yes! It works!
I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
And at last it works as expected on my collection with 5 nodes and about 400 shards.
Original patch was slightly incompatible with 7.6.0
I hope this patch will help to try this feature with 7.6
https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20

RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
But new replica types made it very helpful to keep big clusters in order...

I wonder, why there is no any jira about this case (or maybe I missed it)?
Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
--
Vadim

> -----Original Message-----
> From: Vadim Ivanov [mailto:[hidden email]]
> Sent: Friday, December 07, 2018 6:13 PM
> To: [hidden email]
> Subject: RE: REBALANCELEADERS is not reliable
>
> I'm waiting for 7.6 or 7.5.1 and plan to apply patch from  Endika Posadas to it.
> Then test again and hope it'll help
> --
> Vadim
>
>
> > -----Original Message-----
> > From: Bernd Fehling [mailto:[hidden email]]
> > Sent: Friday, December 07, 2018 12:01 PM
> > To: [hidden email]
> > Subject: Re: REBALANCELEADERS is not reliable
> >
> > Thanks for looking this up.
> > It could be a hint where to jump into the code.
> > I wonder why they rejected a jira ticket about this problem?
> >
> > Regards, Bernd
> >
> > Am 06.12.18 um 16:31 schrieb Vadim Ivanov:
> > > Is solr-dev forum I came across this post
> > > http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-
> > deleted-when-rebalancing-leaders-td4417040.html
> > > May be it will shed some light?
> > >
> > >
> > >> -----Original Message-----
> > >> From: Atita Arora [mailto:[hidden email]]
> > >> Sent: Thursday, November 29, 2018 11:03 PM
> > >> To: [hidden email]
> > >> Subject: Re: REBALANCELEADERS is not reliable
> > >>
> > >> Indeed, I tried that on 7.4 & 7.5 too, indeed did not work for me as well,
> > >> even with the preferredLeader property as recommended in the
> > >> documentation.
> > >> I handled it with a little hack but certainly this dint work as expected.
> > >> I can provide more details if there's a ticket.
> > >>
> > >> On Thu, Nov 29, 2018 at 8:42 PM Aman Tandon
> > >> <[hidden email]> wrote:
> > >>
> > >>> ++ correction
> > >>>
> > >>> On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email]
> > >> wrote:
> > >>>
> > >>>> For me today, I deleted the leader replica of one of the two shard
> > >>>> collection. Then other replicas of that shard wasn't getting elected for
> > >>>> leader.
> > >>>>
> > >>>> After waiting for long tried the setting addreplicaprop preferred leader
> > >>>> on one of the replica then tried FORCELEADER but no luck. Then also
> > tried
> > >>>> rebalance but no help. Finally have to recreate the whole collection.
> > >>>>
> > >>>> Not sure what was the issue but both FORCELEADER AND
> REBALANCING
> > >> didn't
> > >>>> work if there was no leader however preferred leader property was
> > setted.
> > >>>>
> > >>>> On Wed, Nov 28, 2018, 12:54 Bernd Fehling <
> > >>> [hidden email]
> > >>>> wrote:
> > >>>>
> > >>>>> Hi Vadim,
> > >>>>>
> > >>>>> thanks for confirming.
> > >>>>> So it seems to be a general problem with Solr 6.x, 7.x and might
> > >>>>> be still there in the most recent versions.
> > >>>>>
> > >>>>> But where to start to debug this problem, is it something not
> > >>>>> correctly stored in zookeeper or is overseer the problem?
> > >>>>>
> > >>>>> I was also reading something about a "leader queue" where possible
> > >>>>> leaders have to be requeued or something similar.
> > >>>>>
> > >>>>> May be I should try to get a situation where a "locked" core
> > >>>>> is on the overseer and then connect the debugger to it and step
> > >>>>> through it.
> > >>>>> Peeking and poking around, like old Commodore 64 days :-)
> > >>>>>
> > >>>>> Regards, Bernd
> > >>>>>
> > >>>>>
> > >>>>> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
> > >>>>>> Hi, Bernd
> > >>>>>> I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> > >>>>>> I had very similar results and notion that it's not reliable :(
> > >>>>>> --
> > >>>>>> Br, Vadim
> > >>>>>>
> > >>>>>>> -----Original Message-----
> > >>>>>>> From: Bernd Fehling [mailto:[hidden email]]
> > >>>>>>> Sent: Tuesday, November 27, 2018 5:13 PM
> > >>>>>>> To: [hidden email]
> > >>>>>>> Subject: REBALANCELEADERS is not reliable
> > >>>>>>>
> > >>>>>>> Hi list,
> > >>>>>>>
> > >>>>>>> unfortunately REBALANCELEADERS is not reliable and the leader
> > >>>>>>> election has unpredictable results with SolrCloud 6.6.5 and
> > >>>>>>> Zookeeper 3.4.10.
> > >>>>>>> Seen with 5 shards / 3 replicas.
> > >>>>>>>
> > >>>>>>> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> > >>>>>>> - setting with ADDREPLICAPROP the property preferredLeader to
> > other
> > >>>>> replicas
> > >>>>>>> - calling REBALANCELEADERS
> > >>>>>>> - some leaders have changed, some not.
> > >>>>>>>
> > >>>>>>> I then tried:
> > >>>>>>> - removing all preferredLeader properties from replicas which
> > >>>>> succeeded.
> > >>>>>>> - trying again REBALANCELEADERS for the rest. No success.
> > >>>>>>> - Shutting down nodes to force the leader to a specific replica left
> > >>>>> running.
> > >>>>>>>    No success.
> > >>>>>>> - calling REBALANCELEADERS responds that the replica is inactive!!!
> > >>>>>>> - calling CLUSTERSTATUS reports that the replica is active!!!
> > >>>>>>>
> > >>>>>>> Also, the replica which don't want to become leader is not in the
> > >>> list
> > >>>>>>> of collections->[collection_name]->leader_elect->shard1..x-
> >election
> > >>>>>>>
> > >>>>>>> Where is CLUSTERSTATUS getting it's state info from?
> > >>>>>>>
> > >>>>>>> Has anyone else problems with REBALANCELEADERS?
> > >>>>>>>
> > >>>>>>> I noticed that the Reference Guide writes "preferredLeader" (with
> > >>>>> capital "L")
> > >>>>>>> but the JAVA code has "preferredleader".
> > >>>>>>>
> > >>>>>>> Regards, Bernd
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >

Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Erick Erickson
You can go here: https://issues.apache.org/jira, create a signon and
freely create JIRAs. Please attach the patch as well. I hadn't really
thought very carefully about REBALANCELEADERS and the new replica
types, but that does change the use-case.

Best,
Erick

On Thu, Dec 20, 2018 at 3:31 AM Vadim Ivanov
<[hidden email]> wrote:

>
> Yes! It works!
> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> And at last it works as expected on my collection with 5 nodes and about 400 shards.
> Original patch was slightly incompatible with 7.6.0
> I hope this patch will help to try this feature with 7.6
> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
>
> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> But new replica types made it very helpful to keep big clusters in order...
>
> I wonder, why there is no any jira about this case (or maybe I missed it)?
> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
> --
> Vadim
>
> > -----Original Message-----
> > From: Vadim Ivanov [mailto:[hidden email]]
> > Sent: Friday, December 07, 2018 6:13 PM
> > To: [hidden email]
> > Subject: RE: REBALANCELEADERS is not reliable
> >
> > I'm waiting for 7.6 or 7.5.1 and plan to apply patch from  Endika Posadas to it.
> > Then test again and hope it'll help
> > --
> > Vadim
> >
> >
> > > -----Original Message-----
> > > From: Bernd Fehling [mailto:[hidden email]]
> > > Sent: Friday, December 07, 2018 12:01 PM
> > > To: [hidden email]
> > > Subject: Re: REBALANCELEADERS is not reliable
> > >
> > > Thanks for looking this up.
> > > It could be a hint where to jump into the code.
> > > I wonder why they rejected a jira ticket about this problem?
> > >
> > > Regards, Bernd
> > >
> > > Am 06.12.18 um 16:31 schrieb Vadim Ivanov:
> > > > Is solr-dev forum I came across this post
> > > > http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-
> > > deleted-when-rebalancing-leaders-td4417040.html
> > > > May be it will shed some light?
> > > >
> > > >
> > > >> -----Original Message-----
> > > >> From: Atita Arora [mailto:[hidden email]]
> > > >> Sent: Thursday, November 29, 2018 11:03 PM
> > > >> To: [hidden email]
> > > >> Subject: Re: REBALANCELEADERS is not reliable
> > > >>
> > > >> Indeed, I tried that on 7.4 & 7.5 too, indeed did not work for me as well,
> > > >> even with the preferredLeader property as recommended in the
> > > >> documentation.
> > > >> I handled it with a little hack but certainly this dint work as expected.
> > > >> I can provide more details if there's a ticket.
> > > >>
> > > >> On Thu, Nov 29, 2018 at 8:42 PM Aman Tandon
> > > >> <[hidden email]> wrote:
> > > >>
> > > >>> ++ correction
> > > >>>
> > > >>> On Fri, Nov 30, 2018, 01:10 Aman Tandon <[hidden email]
> > > >> wrote:
> > > >>>
> > > >>>> For me today, I deleted the leader replica of one of the two shard
> > > >>>> collection. Then other replicas of that shard wasn't getting elected for
> > > >>>> leader.
> > > >>>>
> > > >>>> After waiting for long tried the setting addreplicaprop preferred leader
> > > >>>> on one of the replica then tried FORCELEADER but no luck. Then also
> > > tried
> > > >>>> rebalance but no help. Finally have to recreate the whole collection.
> > > >>>>
> > > >>>> Not sure what was the issue but both FORCELEADER AND
> > REBALANCING
> > > >> didn't
> > > >>>> work if there was no leader however preferred leader property was
> > > setted.
> > > >>>>
> > > >>>> On Wed, Nov 28, 2018, 12:54 Bernd Fehling <
> > > >>> [hidden email]
> > > >>>> wrote:
> > > >>>>
> > > >>>>> Hi Vadim,
> > > >>>>>
> > > >>>>> thanks for confirming.
> > > >>>>> So it seems to be a general problem with Solr 6.x, 7.x and might
> > > >>>>> be still there in the most recent versions.
> > > >>>>>
> > > >>>>> But where to start to debug this problem, is it something not
> > > >>>>> correctly stored in zookeeper or is overseer the problem?
> > > >>>>>
> > > >>>>> I was also reading something about a "leader queue" where possible
> > > >>>>> leaders have to be requeued or something similar.
> > > >>>>>
> > > >>>>> May be I should try to get a situation where a "locked" core
> > > >>>>> is on the overseer and then connect the debugger to it and step
> > > >>>>> through it.
> > > >>>>> Peeking and poking around, like old Commodore 64 days :-)
> > > >>>>>
> > > >>>>> Regards, Bernd
> > > >>>>>
> > > >>>>>
> > > >>>>> Am 27.11.18 um 15:47 schrieb Vadim Ivanov:
> > > >>>>>> Hi, Bernd
> > > >>>>>> I have tried REBALANCELEADERS with Solr 6.3 and 7.5
> > > >>>>>> I had very similar results and notion that it's not reliable :(
> > > >>>>>> --
> > > >>>>>> Br, Vadim
> > > >>>>>>
> > > >>>>>>> -----Original Message-----
> > > >>>>>>> From: Bernd Fehling [mailto:[hidden email]]
> > > >>>>>>> Sent: Tuesday, November 27, 2018 5:13 PM
> > > >>>>>>> To: [hidden email]
> > > >>>>>>> Subject: REBALANCELEADERS is not reliable
> > > >>>>>>>
> > > >>>>>>> Hi list,
> > > >>>>>>>
> > > >>>>>>> unfortunately REBALANCELEADERS is not reliable and the leader
> > > >>>>>>> election has unpredictable results with SolrCloud 6.6.5 and
> > > >>>>>>> Zookeeper 3.4.10.
> > > >>>>>>> Seen with 5 shards / 3 replicas.
> > > >>>>>>>
> > > >>>>>>> - CLUSTERSTATUS reports all replicas (core_nodes) as state=active.
> > > >>>>>>> - setting with ADDREPLICAPROP the property preferredLeader to
> > > other
> > > >>>>> replicas
> > > >>>>>>> - calling REBALANCELEADERS
> > > >>>>>>> - some leaders have changed, some not.
> > > >>>>>>>
> > > >>>>>>> I then tried:
> > > >>>>>>> - removing all preferredLeader properties from replicas which
> > > >>>>> succeeded.
> > > >>>>>>> - trying again REBALANCELEADERS for the rest. No success.
> > > >>>>>>> - Shutting down nodes to force the leader to a specific replica left
> > > >>>>> running.
> > > >>>>>>>    No success.
> > > >>>>>>> - calling REBALANCELEADERS responds that the replica is inactive!!!
> > > >>>>>>> - calling CLUSTERSTATUS reports that the replica is active!!!
> > > >>>>>>>
> > > >>>>>>> Also, the replica which don't want to become leader is not in the
> > > >>> list
> > > >>>>>>> of collections->[collection_name]->leader_elect->shard1..x-
> > >election
> > > >>>>>>>
> > > >>>>>>> Where is CLUSTERSTATUS getting it's state info from?
> > > >>>>>>>
> > > >>>>>>> Has anyone else problems with REBALANCELEADERS?
> > > >>>>>>>
> > > >>>>>>> I noticed that the Reference Guide writes "preferredLeader" (with
> > > >>>>> capital "L")
> > > >>>>>>> but the JAVA code has "preferredleader".
> > > >>>>>>>
> > > >>>>>>> Regards, Bernd
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>>
> > > >
>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Bernd Fehling
In reply to this post by Vadim Ivanov
Hi Vadim,
I just tried it with 6.6.5.
In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
one shard to become leader. But noticed that one shard already was
leader. No errors or exceptions in logs.
May be I should enable debug logging and try again to see all logging
messages from the patch.

Might be they also changed other parts between 6.6.5 and 7.6.0 so that
it works for you.

I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.

Regards, Bernd


Am 20.12.18 um 12:31 schrieb Vadim Ivanov:

> Yes! It works!
> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> And at last it works as expected on my collection with 5 nodes and about 400 shards.
> Original patch was slightly incompatible with 7.6.0
> I hope this patch will help to try this feature with 7.6
> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
>
> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> But new replica types made it very helpful to keep big clusters in order...
>
> I wonder, why there is no any jira about this case (or maybe I missed it)?
> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Erick Erickson
Nothing that I know of was _intentionally_ changed with this between
6x and 7x. That said, nothing that I know of was done to verify that
TLOG and PULL replicas (added in 7x) were handled correctly. There's a
test "TestRebalanceLeaders" for this functionality that has run since
the feature was put in, but it has _not_ been modified to create TLOG
and PULL replicas and test with those.

For this patch to be complete, we should either extend that test or
make another that fails without this patch and succeeds with it.

I'd probably recommend modifying TestRebalanceLeaders to randomly
create TLOG and (maybe) PULL replicas so we'd keep covering the
various cases.

Best,
Erick


On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
<[hidden email]> wrote:

>
> Hi Vadim,
> I just tried it with 6.6.5.
> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
> one shard to become leader. But noticed that one shard already was
> leader. No errors or exceptions in logs.
> May be I should enable debug logging and try again to see all logging
> messages from the patch.
>
> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
> it works for you.
>
> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
>
> Regards, Bernd
>
>
> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
> > Yes! It works!
> > I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> > And at last it works as expected on my collection with 5 nodes and about 400 shards.
> > Original patch was slightly incompatible with 7.6.0
> > I hope this patch will help to try this feature with 7.6
> > https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
> >
> > RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> > But new replica types made it very helpful to keep big clusters in order...
> >
> > I wonder, why there is no any jira about this case (or maybe I missed it)?
> > Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
> >
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Erick Erickson
I'm reworking the test case, so hold off on doing that. If you want to
raise a JIRA, though. please do and attach your patch...

On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:

>
> Nothing that I know of was _intentionally_ changed with this between
> 6x and 7x. That said, nothing that I know of was done to verify that
> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
> test "TestRebalanceLeaders" for this functionality that has run since
> the feature was put in, but it has _not_ been modified to create TLOG
> and PULL replicas and test with those.
>
> For this patch to be complete, we should either extend that test or
> make another that fails without this patch and succeeds with it.
>
> I'd probably recommend modifying TestRebalanceLeaders to randomly
> create TLOG and (maybe) PULL replicas so we'd keep covering the
> various cases.
>
> Best,
> Erick
>
>
> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
> <[hidden email]> wrote:
> >
> > Hi Vadim,
> > I just tried it with 6.6.5.
> > In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
> > one shard to become leader. But noticed that one shard already was
> > leader. No errors or exceptions in logs.
> > May be I should enable debug logging and try again to see all logging
> > messages from the patch.
> >
> > Might be they also changed other parts between 6.6.5 and 7.6.0 so that
> > it works for you.
> >
> > I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
> > even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
> >
> > Regards, Bernd
> >
> >
> > Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
> > > Yes! It works!
> > > I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> > > And at last it works as expected on my collection with 5 nodes and about 400 shards.
> > > Original patch was slightly incompatible with 7.6.0
> > > I hope this patch will help to try this feature with 7.6
> > > https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
> > >
> > > RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> > > But new replica types made it very helpful to keep big clusters in order...
> > >
> > > I wonder, why there is no any jira about this case (or maybe I missed it)?
> > > Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
> > >
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Bernd Fehling
In reply to this post by Erick Erickson
As far as I could see with debugger there is still a problem in requeing.

There is a watcher and it is recognized that the watcher is not a preferredleader.
So it tries to locate a preferredleader with success.
It then calls makeReplicaFirstWatcher and gets a new sequence number for
the preferredleader replica. But now we have two replicas with the same
sequence number. One replica which already owns that sequence number and
the replica which got the new (and the same) number as new sequence number.
It now tries to solve this with queueNodesWithSameSequence.
Might be something in rejoinElection.
At least the call to rejoinElection seems right. For preferredleader it
is true for rejoinAtHead and for the other replica with same sequence number
it is false for rejoinAtHead.

A test case should have 3 shards with 3 cores per shard and should try to
set preferredleader to different replicas at random. And then try to
rebalance and check the results.

So far, regards, Bernd


Am 21.12.18 um 07:11 schrieb Erick Erickson:

> I'm reworking the test case, so hold off on doing that. If you want to
> raise a JIRA, though. please do and attach your patch...
>
> On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:
>>
>> Nothing that I know of was _intentionally_ changed with this between
>> 6x and 7x. That said, nothing that I know of was done to verify that
>> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
>> test "TestRebalanceLeaders" for this functionality that has run since
>> the feature was put in, but it has _not_ been modified to create TLOG
>> and PULL replicas and test with those.
>>
>> For this patch to be complete, we should either extend that test or
>> make another that fails without this patch and succeeds with it.
>>
>> I'd probably recommend modifying TestRebalanceLeaders to randomly
>> create TLOG and (maybe) PULL replicas so we'd keep covering the
>> various cases.
>>
>> Best,
>> Erick
>>
>>
>> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
>> <[hidden email]> wrote:
>>>
>>> Hi Vadim,
>>> I just tried it with 6.6.5.
>>> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
>>> one shard to become leader. But noticed that one shard already was
>>> leader. No errors or exceptions in logs.
>>> May be I should enable debug logging and try again to see all logging
>>> messages from the patch.
>>>
>>> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
>>> it works for you.
>>>
>>> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
>>> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
>>>
>>> Regards, Bernd
>>>
>>>
>>> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
>>>> Yes! It works!
>>>> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
>>>> And at last it works as expected on my collection with 5 nodes and about 400 shards.
>>>> Original patch was slightly incompatible with 7.6.0
>>>> I hope this patch will help to try this feature with 7.6
>>>> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
>>>>
>>>> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
>>>> But new replica types made it very helpful to keep big clusters in order...
>>>>
>>>> I wonder, why there is no any jira about this case (or maybe I missed it)?
>>>> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
>>>>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Erick Erickson
I looked at the test last night and it's...disturbing. It succeeds
100% of the time. Manual testing seems to fail very often.
Of course it was late and I was a bit cross-eyed, so maybe
I wasn't looking at the manual tests correctly. Or maybe the
test is buggy.

I beasted the test 100x last night and all of them succeeded.

This was with all NRT replicas.

Today I'm going to modify the test into a stand-alone program
to see if it's something in the test environment that causes
it to succeed. I've got to get this to fail as a unit test before I
have confidence in any fixes, and also confidence that things
like this will be caught going forward.

Erick

On Fri, Dec 21, 2018 at 3:59 AM Bernd Fehling
<[hidden email]> wrote:

>
> As far as I could see with debugger there is still a problem in requeing.
>
> There is a watcher and it is recognized that the watcher is not a preferredleader.
> So it tries to locate a preferredleader with success.
> It then calls makeReplicaFirstWatcher and gets a new sequence number for
> the preferredleader replica. But now we have two replicas with the same
> sequence number. One replica which already owns that sequence number and
> the replica which got the new (and the same) number as new sequence number.
> It now tries to solve this with queueNodesWithSameSequence.
> Might be something in rejoinElection.
> At least the call to rejoinElection seems right. For preferredleader it
> is true for rejoinAtHead and for the other replica with same sequence number
> it is false for rejoinAtHead.
>
> A test case should have 3 shards with 3 cores per shard and should try to
> set preferredleader to different replicas at random. And then try to
> rebalance and check the results.
>
> So far, regards, Bernd
>
>
> Am 21.12.18 um 07:11 schrieb Erick Erickson:
> > I'm reworking the test case, so hold off on doing that. If you want to
> > raise a JIRA, though. please do and attach your patch...
> >
> > On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:
> >>
> >> Nothing that I know of was _intentionally_ changed with this between
> >> 6x and 7x. That said, nothing that I know of was done to verify that
> >> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
> >> test "TestRebalanceLeaders" for this functionality that has run since
> >> the feature was put in, but it has _not_ been modified to create TLOG
> >> and PULL replicas and test with those.
> >>
> >> For this patch to be complete, we should either extend that test or
> >> make another that fails without this patch and succeeds with it.
> >>
> >> I'd probably recommend modifying TestRebalanceLeaders to randomly
> >> create TLOG and (maybe) PULL replicas so we'd keep covering the
> >> various cases.
> >>
> >> Best,
> >> Erick
> >>
> >>
> >> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
> >> <[hidden email]> wrote:
> >>>
> >>> Hi Vadim,
> >>> I just tried it with 6.6.5.
> >>> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
> >>> one shard to become leader. But noticed that one shard already was
> >>> leader. No errors or exceptions in logs.
> >>> May be I should enable debug logging and try again to see all logging
> >>> messages from the patch.
> >>>
> >>> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
> >>> it works for you.
> >>>
> >>> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
> >>> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
> >>>
> >>> Regards, Bernd
> >>>
> >>>
> >>> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
> >>>> Yes! It works!
> >>>> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> >>>> And at last it works as expected on my collection with 5 nodes and about 400 shards.
> >>>> Original patch was slightly incompatible with 7.6.0
> >>>> I hope this patch will help to try this feature with 7.6
> >>>> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
> >>>>
> >>>> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> >>>> But new replica types made it very helpful to keep big clusters in order...
> >>>>
> >>>> I wonder, why there is no any jira about this case (or maybe I missed it)?
> >>>> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
> >>>>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Bernd Fehling
In reply to this post by Bernd Fehling
Hi Erick,

after some more hours of debugging the rough result is, who ever invented
this leader election did not check if an action returns the estimated
result. There are only checks for exceptions, true/false, new sequence
numbers and so on, but never if a leader election to the preferredleader
really took place.

If doing a rebalanceleaders to preferredleader I also have to check if:
- a rebalance took place
- the preferredleader has really become leader (and not anyone else)

Currently this is not checked and the call rebalanceleaders to preferredleader
is like a shot into the dark with hope of success. And thats why any
problems have never been discovered or reported.

Bernd


Am 21.12.18 um 18:00 schrieb Erick Erickson:

> I looked at the test last night and it's...disturbing. It succeeds
> 100% of the time. Manual testing seems to fail very often.
> Of course it was late and I was a bit cross-eyed, so maybe
> I wasn't looking at the manual tests correctly. Or maybe the
> test is buggy.
>
> I beasted the test 100x last night and all of them succeeded.
>
> This was with all NRT replicas.
>
> Today I'm going to modify the test into a stand-alone program
> to see if it's something in the test environment that causes
> it to succeed. I've got to get this to fail as a unit test before I
> have confidence in any fixes, and also confidence that things
> like this will be caught going forward.
>
> Erick
>
> On Fri, Dec 21, 2018 at 3:59 AM Bernd Fehling
> <[hidden email]> wrote:
>>
>> As far as I could see with debugger there is still a problem in requeing.
>>
>> There is a watcher and it is recognized that the watcher is not a preferredleader.
>> So it tries to locate a preferredleader with success.
>> It then calls makeReplicaFirstWatcher and gets a new sequence number for
>> the preferredleader replica. But now we have two replicas with the same
>> sequence number. One replica which already owns that sequence number and
>> the replica which got the new (and the same) number as new sequence number.
>> It now tries to solve this with queueNodesWithSameSequence.
>> Might be something in rejoinElection.
>> At least the call to rejoinElection seems right. For preferredleader it
>> is true for rejoinAtHead and for the other replica with same sequence number
>> it is false for rejoinAtHead.
>>
>> A test case should have 3 shards with 3 cores per shard and should try to
>> set preferredleader to different replicas at random. And then try to
>> rebalance and check the results.
>>
>> So far, regards, Bernd
>>
>>
>> Am 21.12.18 um 07:11 schrieb Erick Erickson:
>>> I'm reworking the test case, so hold off on doing that. If you want to
>>> raise a JIRA, though. please do and attach your patch...
>>>
>>> On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:
>>>>
>>>> Nothing that I know of was _intentionally_ changed with this between
>>>> 6x and 7x. That said, nothing that I know of was done to verify that
>>>> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
>>>> test "TestRebalanceLeaders" for this functionality that has run since
>>>> the feature was put in, but it has _not_ been modified to create TLOG
>>>> and PULL replicas and test with those.
>>>>
>>>> For this patch to be complete, we should either extend that test or
>>>> make another that fails without this patch and succeeds with it.
>>>>
>>>> I'd probably recommend modifying TestRebalanceLeaders to randomly
>>>> create TLOG and (maybe) PULL replicas so we'd keep covering the
>>>> various cases.
>>>>
>>>> Best,
>>>> Erick
>>>>
>>>>
>>>> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
>>>> <[hidden email]> wrote:
>>>>>
>>>>> Hi Vadim,
>>>>> I just tried it with 6.6.5.
>>>>> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
>>>>> one shard to become leader. But noticed that one shard already was
>>>>> leader. No errors or exceptions in logs.
>>>>> May be I should enable debug logging and try again to see all logging
>>>>> messages from the patch.
>>>>>
>>>>> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
>>>>> it works for you.
>>>>>
>>>>> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
>>>>> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
>>>>>
>>>>> Regards, Bernd
>>>>>
>>>>>
>>>>> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
>>>>>> Yes! It works!
>>>>>> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
>>>>>> And at last it works as expected on my collection with 5 nodes and about 400 shards.
>>>>>> Original patch was slightly incompatible with 7.6.0
>>>>>> I hope this patch will help to try this feature with 7.6
>>>>>> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
>>>>>>
>>>>>> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
>>>>>> But new replica types made it very helpful to keep big clusters in order...
>>>>>>
>>>>>> I wonder, why there is no any jira about this case (or maybe I missed it)?
>>>>>> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
>>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Erick Erickson
It's weirder than that. In the current test on master, the
assumption is that the node recorded as leader in ZK
is actually the leader, see
TestRebalanceLeaders.checkZkLeadersAgree(). The theory
is that the identified leader node in ZK is actually the leader
after the rebalance command. But you're right, I don't see
an actual check that the collection's status agrees.

That aside, though, there are several problems I'm uncovering

1> BALANCESHARDUNIQUE can wind up with multiple
"preferredLeader" properties defined. Some time between
the original code and now someone refactored a bunch of
code and missed removing a unique property if it was
already assigned and being assigned to another replica
in the same slice.

2> to make it much worse, I've rewritten the tests
extensively and I can beast the rewritten tests 1,000
times and no failures. If I test manually by just issuing
the commands, everything works fine. By "testing manually"
I mean (working with 4 Vms, 10 shards 4 replicas)
> create the collection
> issue the BALANCESHARDUNIQUE command
> issue the REBALANCELEADERS command


However, if instead I
> create the collection
> issue the BALANCESHARDUNIQUE command
> shut down 3 of 4 Solr instances so all the leaders
   are on the same host.
> restart the 3 instances
> issue the REBALANCELEADERS command then
   it doesn't work.

At least that's what I think I'm seeing, but it makes no
real sense yet.

So I'm first trying to understand why my manual test
fails so regularly, then I can incorporate that setup
into the unit test (I'm thinking of just shutting down
and restarting some of the Jetty instances).

But it's a total mystery to me why restarting Solr instances
should have any effect. But that's certainly not
something that happens in the current test so I have
hopes that tracking that down will lead to understanding
what the invalid assumption I'm making is and we can
test for that too.,

On Tue, Jan 8, 2019 at 1:42 AM Bernd Fehling
<[hidden email]> wrote:

>
> Hi Erick,
>
> after some more hours of debugging the rough result is, who ever invented
> this leader election did not check if an action returns the estimated
> result. There are only checks for exceptions, true/false, new sequence
> numbers and so on, but never if a leader election to the preferredleader
> really took place.
>
> If doing a rebalanceleaders to preferredleader I also have to check if:
> - a rebalance took place
> - the preferredleader has really become leader (and not anyone else)
>
> Currently this is not checked and the call rebalanceleaders to preferredleader
> is like a shot into the dark with hope of success. And thats why any
> problems have never been discovered or reported.
>
> Bernd
>
>
> Am 21.12.18 um 18:00 schrieb Erick Erickson:
> > I looked at the test last night and it's...disturbing. It succeeds
> > 100% of the time. Manual testing seems to fail very often.
> > Of course it was late and I was a bit cross-eyed, so maybe
> > I wasn't looking at the manual tests correctly. Or maybe the
> > test is buggy.
> >
> > I beasted the test 100x last night and all of them succeeded.
> >
> > This was with all NRT replicas.
> >
> > Today I'm going to modify the test into a stand-alone program
> > to see if it's something in the test environment that causes
> > it to succeed. I've got to get this to fail as a unit test before I
> > have confidence in any fixes, and also confidence that things
> > like this will be caught going forward.
> >
> > Erick
> >
> > On Fri, Dec 21, 2018 at 3:59 AM Bernd Fehling
> > <[hidden email]> wrote:
> >>
> >> As far as I could see with debugger there is still a problem in requeing.
> >>
> >> There is a watcher and it is recognized that the watcher is not a preferredleader.
> >> So it tries to locate a preferredleader with success.
> >> It then calls makeReplicaFirstWatcher and gets a new sequence number for
> >> the preferredleader replica. But now we have two replicas with the same
> >> sequence number. One replica which already owns that sequence number and
> >> the replica which got the new (and the same) number as new sequence number.
> >> It now tries to solve this with queueNodesWithSameSequence.
> >> Might be something in rejoinElection.
> >> At least the call to rejoinElection seems right. For preferredleader it
> >> is true for rejoinAtHead and for the other replica with same sequence number
> >> it is false for rejoinAtHead.
> >>
> >> A test case should have 3 shards with 3 cores per shard and should try to
> >> set preferredleader to different replicas at random. And then try to
> >> rebalance and check the results.
> >>
> >> So far, regards, Bernd
> >>
> >>
> >> Am 21.12.18 um 07:11 schrieb Erick Erickson:
> >>> I'm reworking the test case, so hold off on doing that. If you want to
> >>> raise a JIRA, though. please do and attach your patch...
> >>>
> >>> On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:
> >>>>
> >>>> Nothing that I know of was _intentionally_ changed with this between
> >>>> 6x and 7x. That said, nothing that I know of was done to verify that
> >>>> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
> >>>> test "TestRebalanceLeaders" for this functionality that has run since
> >>>> the feature was put in, but it has _not_ been modified to create TLOG
> >>>> and PULL replicas and test with those.
> >>>>
> >>>> For this patch to be complete, we should either extend that test or
> >>>> make another that fails without this patch and succeeds with it.
> >>>>
> >>>> I'd probably recommend modifying TestRebalanceLeaders to randomly
> >>>> create TLOG and (maybe) PULL replicas so we'd keep covering the
> >>>> various cases.
> >>>>
> >>>> Best,
> >>>> Erick
> >>>>
> >>>>
> >>>> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
> >>>> <[hidden email]> wrote:
> >>>>>
> >>>>> Hi Vadim,
> >>>>> I just tried it with 6.6.5.
> >>>>> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
> >>>>> one shard to become leader. But noticed that one shard already was
> >>>>> leader. No errors or exceptions in logs.
> >>>>> May be I should enable debug logging and try again to see all logging
> >>>>> messages from the patch.
> >>>>>
> >>>>> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
> >>>>> it works for you.
> >>>>>
> >>>>> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
> >>>>> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
> >>>>>
> >>>>> Regards, Bernd
> >>>>>
> >>>>>
> >>>>> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
> >>>>>> Yes! It works!
> >>>>>> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> >>>>>> And at last it works as expected on my collection with 5 nodes and about 400 shards.
> >>>>>> Original patch was slightly incompatible with 7.6.0
> >>>>>> I hope this patch will help to try this feature with 7.6
> >>>>>> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
> >>>>>>
> >>>>>> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> >>>>>> But new replica types made it very helpful to keep big clusters in order...
> >>>>>>
> >>>>>> I wonder, why there is no any jira about this case (or maybe I missed it)?
> >>>>>> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
> >>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Bernd Fehling
In reply to this post by Bernd Fehling
Yes, your findings are also very strange.
I wonder if we can discover the "inventor" of all this and ask him
how it should work or better how he originally wanted it to work.

Comments in the code (RebalanceLeaders.java) state that it is possible
to have more than one electionNode with the same sequence number.
Absolutely strange.

I wonder why the queue is not rotated until the new and preferred
leader is at front (position 0)?
But why is it a queue anyway?
Wherever I see any java code to get the content from the queue it
is sorted. Where is the sense of this?

Also, the elctionNodes have another attribute with name "ephemeral".
Where is that for and why is it not tested in TestRebalanceLeaders.java?

Regards, Bernd


Am 09.01.19 um 02:31 schrieb Erick Erickson:

> It's weirder than that. In the current test on master, the
> assumption is that the node recorded as leader in ZK
> is actually the leader, see
> TestRebalanceLeaders.checkZkLeadersAgree(). The theory
> is that the identified leader node in ZK is actually the leader
> after the rebalance command. But you're right, I don't see
> an actual check that the collection's status agrees.
>
> That aside, though, there are several problems I'm uncovering
>
> 1> BALANCESHARDUNIQUE can wind up with multiple
> "preferredLeader" properties defined. Some time between
> the original code and now someone refactored a bunch of
> code and missed removing a unique property if it was
> already assigned and being assigned to another replica
> in the same slice.
>
> 2> to make it much worse, I've rewritten the tests
> extensively and I can beast the rewritten tests 1,000
> times and no failures. If I test manually by just issuing
> the commands, everything works fine. By "testing manually"
> I mean (working with 4 Vms, 10 shards 4 replicas)
>> create the collection
>> issue the BALANCESHARDUNIQUE command
>> issue the REBALANCELEADERS command
>
>
> However, if instead I
>> create the collection
>> issue the BALANCESHARDUNIQUE command
>> shut down 3 of 4 Solr instances so all the leaders
>     are on the same host.
>> restart the 3 instances
>> issue the REBALANCELEADERS command then
>     it doesn't work.
>
> At least that's what I think I'm seeing, but it makes no
> real sense yet.
>
> So I'm first trying to understand why my manual test
> fails so regularly, then I can incorporate that setup
> into the unit test (I'm thinking of just shutting down
> and restarting some of the Jetty instances).
>
> But it's a total mystery to me why restarting Solr instances
> should have any effect. But that's certainly not
> something that happens in the current test so I have
> hopes that tracking that down will lead to understanding
> what the invalid assumption I'm making is and we can
> test for that too.,
>
> On Tue, Jan 8, 2019 at 1:42 AM Bernd Fehling
> <[hidden email]> wrote:
>>
>> Hi Erick,
>>
>> after some more hours of debugging the rough result is, who ever invented
>> this leader election did not check if an action returns the estimated
>> result. There are only checks for exceptions, true/false, new sequence
>> numbers and so on, but never if a leader election to the preferredleader
>> really took place.
>>
>> If doing a rebalanceleaders to preferredleader I also have to check if:
>> - a rebalance took place
>> - the preferredleader has really become leader (and not anyone else)
>>
>> Currently this is not checked and the call rebalanceleaders to preferredleader
>> is like a shot into the dark with hope of success. And thats why any
>> problems have never been discovered or reported.
>>
>> Bernd
>>
>>
>> Am 21.12.18 um 18:00 schrieb Erick Erickson:
>>> I looked at the test last night and it's...disturbing. It succeeds
>>> 100% of the time. Manual testing seems to fail very often.
>>> Of course it was late and I was a bit cross-eyed, so maybe
>>> I wasn't looking at the manual tests correctly. Or maybe the
>>> test is buggy.
>>>
>>> I beasted the test 100x last night and all of them succeeded.
>>>
>>> This was with all NRT replicas.
>>>
>>> Today I'm going to modify the test into a stand-alone program
>>> to see if it's something in the test environment that causes
>>> it to succeed. I've got to get this to fail as a unit test before I
>>> have confidence in any fixes, and also confidence that things
>>> like this will be caught going forward.
>>>
>>> Erick
>>>
>>> On Fri, Dec 21, 2018 at 3:59 AM Bernd Fehling
>>> <[hidden email]> wrote:
>>>>
>>>> As far as I could see with debugger there is still a problem in requeing.
>>>>
>>>> There is a watcher and it is recognized that the watcher is not a preferredleader.
>>>> So it tries to locate a preferredleader with success.
>>>> It then calls makeReplicaFirstWatcher and gets a new sequence number for
>>>> the preferredleader replica. But now we have two replicas with the same
>>>> sequence number. One replica which already owns that sequence number and
>>>> the replica which got the new (and the same) number as new sequence number.
>>>> It now tries to solve this with queueNodesWithSameSequence.
>>>> Might be something in rejoinElection.
>>>> At least the call to rejoinElection seems right. For preferredleader it
>>>> is true for rejoinAtHead and for the other replica with same sequence number
>>>> it is false for rejoinAtHead.
>>>>
>>>> A test case should have 3 shards with 3 cores per shard and should try to
>>>> set preferredleader to different replicas at random. And then try to
>>>> rebalance and check the results.
>>>>
>>>> So far, regards, Bernd
>>>>
>>>>
>>>> Am 21.12.18 um 07:11 schrieb Erick Erickson:
>>>>> I'm reworking the test case, so hold off on doing that. If you want to
>>>>> raise a JIRA, though. please do and attach your patch...
>>>>>
>>>>> On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:
>>>>>>
>>>>>> Nothing that I know of was _intentionally_ changed with this between
>>>>>> 6x and 7x. That said, nothing that I know of was done to verify that
>>>>>> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
>>>>>> test "TestRebalanceLeaders" for this functionality that has run since
>>>>>> the feature was put in, but it has _not_ been modified to create TLOG
>>>>>> and PULL replicas and test with those.
>>>>>>
>>>>>> For this patch to be complete, we should either extend that test or
>>>>>> make another that fails without this patch and succeeds with it.
>>>>>>
>>>>>> I'd probably recommend modifying TestRebalanceLeaders to randomly
>>>>>> create TLOG and (maybe) PULL replicas so we'd keep covering the
>>>>>> various cases.
>>>>>>
>>>>>> Best,
>>>>>> Erick
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
>>>>>> <[hidden email]> wrote:
>>>>>>>
>>>>>>> Hi Vadim,
>>>>>>> I just tried it with 6.6.5.
>>>>>>> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
>>>>>>> one shard to become leader. But noticed that one shard already was
>>>>>>> leader. No errors or exceptions in logs.
>>>>>>> May be I should enable debug logging and try again to see all logging
>>>>>>> messages from the patch.
>>>>>>>
>>>>>>> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
>>>>>>> it works for you.
>>>>>>>
>>>>>>> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
>>>>>>> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
>>>>>>>
>>>>>>> Regards, Bernd
>>>>>>>
>>>>>>>
>>>>>>> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
>>>>>>>> Yes! It works!
>>>>>>>> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
>>>>>>>> And at last it works as expected on my collection with 5 nodes and about 400 shards.
>>>>>>>> Original patch was slightly incompatible with 7.6.0
>>>>>>>> I hope this patch will help to try this feature with 7.6
>>>>>>>> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
>>>>>>>>
>>>>>>>> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
>>>>>>>> But new replica types made it very helpful to keep big clusters in order...
>>>>>>>>
>>>>>>>> I wonder, why there is no any jira about this case (or maybe I missed it)?
>>>>>>>> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
>>>>>>>>
Reply | Threaded
Open this post in threaded view
|

Re: REBALANCELEADERS is not reliable

Erick Erickson
Executive summary:

The central problem is "how can I insert an ephemeral node
in a specific place in a ZK queue". The code could be much,
much simpler if there were a reliable way to do just that. I haven't
looked at more recent ZKs to see if it's possible, I'd love it if
there were a better way.

On to details:

bq.  wonder if we can discover the "inventor" of all this and ask him
how it should work

Yeah, I can contact that clown. That would be me ;)

The way leader election works is a  ZK recipe where each
ephemeral node only watches the one in front of it. When a
node is deleted, the one watching it is notified.

So let's say we have nodes in this order: 1 watched by 2
watched by 3... In this case 1 is the leader. If 2 should
disappear, 3 gets notified and now watches 1. Nothing
else really happens.

But if the nodes are 1, 2, 3 and 1 disappears 2 gets notified
and says, in effect "I'm first in line so I will be leader". 3
doesn't get any notification.

bq. I wonder why the queue is not rotated until the new and preferred
leader is at front (position 0)

Because then you'd have N leader changes where N is the number
of nodes between the preferred leader and the actual leader in
the leader election queue at the start. That could result in 100s of
changes when only 1 is desired. The original case for this was
exactly that, there could be 100s of shards and several tens
to 100s of replicas.

Hmmmm, I suppose you wouldn't have that many leader changes if
you sent the ephemeral node in the _second_ position to the end
of the queue until the preferredLeader became the one in the second
position. You'd still have a watch fired for every requeueing though. I
haven't measured the cost there. That would also be an added
burden for the Overseer, which has been overwhelmed in the past.

I'm not against that solution, I don't have any real data to
evaluate.

bq. ....is possible to have more than one electionNode with the
same sequence number.

Yeah, and it causes complexity, but I don't have a good way around it. This is
related to your sorting question. ZK itself has a simple ordering, sequential
sequence numbers. Having two the same is the only way I could see (actually I
patterned it off some other code) to insert an ephemeral node second. What you
have then is two nodes "tied" for second by having the same sequence numbers.

Which one ZK thinks is second (and
thus which one becomes the leader if the zero'th ephemeral node disappears)
is based on sorting which includes the session ID, so there's code in there that
has to deal with sending the non-preferred node that's tied for second to the
end of the queue. That's the code that got changed during various
refactorings that I didn't take part in, and the code that's messed up.

bq. Wherever I see any java code to get the content from the queue it
is sorted. Where is the sense of this?

This is implied by the above, but explicitly so the Java code can see the queue
in the same order that ZK does and "do the right thing". In this case
assign the preferred leader's ephemeral node with the same sequence number
that the current second-in-line has and move the current second-in-line to the
end of the queue.

All that said, this code was written several years ago and I haven't looked at
whether there are better methods available now. The actions that are necessary
are:

1> ensure that the preferredLeader is the only node watching the leader in the
    leader election queue

2> re-queue the leader at the end of the leader election queue. Since we'd be
sure the preferredLeader is watching the leader, this action would elect
the proper node as the leader.

Hmmm, note to self. It would help folks in the future if I, you know, documented
those two points in the code. Siiggghhh.

Last night I found what I  _think_ is the problem I was having. Note that the
current code has three problems. I think I have fixes for all of them:

1> assigning the preferredLeader (or any SHARDUNIQUE property) does not
     properly remove that property from other replicas in the shard if
    present. So you may have multiple preferredLeaders in a shard.

2> the code to resolve tied sequence numbers had been changed
     during some refactoring so the wrong node could be elected.

3> the response from the rebalanceleaders command isn't very useful, it's
     on my plate to fix that. Partly it was not reporting useful
     info, and partly your comment from the other day that it returns
     without verifying the leadership has actually changed is well taken. At
     present, it just changes the election queue and assumes that the
     right thing happens. The test code was supposed to point out when
     that assumption was incorrect, but you know the story there.

Currently, the code is pretty ugly in terms of all the junk I put in trying to
track this down, but when I clean it up I'll put up a patch. I added some
code to restart some of the jettys in the test (it's now "@Slow:") that
catches the restart case. Additionally, I changed the test to force
unique properties to be concentrated on a particular node then issue the
BALANCESHARDUNIQUE command to make sure that <1> above
doesn't happen.

Meanwhile, if there's an alternative approach that's simpler I'd be all
for it.

Best,
Erick

On Wed, Jan 9, 2019 at 1:32 AM Bernd Fehling
<[hidden email]> wrote:

>
> Yes, your findings are also very strange.
> I wonder if we can discover the "inventor" of all this and ask him
> how it should work or better how he originally wanted it to work.
>
> Comments in the code (RebalanceLeaders.java) state that it is possible
> to have more than one electionNode with the same sequence number.
> Absolutely strange.
>
> I wonder why the queue is not rotated until the new and preferred
> leader is at front (position 0)?
> But why is it a queue anyway?
> Wherever I see any java code to get the content from the queue it
> is sorted. Where is the sense of this?
>
> Also, the elctionNodes have another attribute with name "ephemeral".
> Where is that for and why is it not tested in TestRebalanceLeaders.java?
>
> Regards, Bernd
>
>
> Am 09.01.19 um 02:31 schrieb Erick Erickson:
> > It's weirder than that. In the current test on master, the
> > assumption is that the node recorded as leader in ZK
> > is actually the leader, see
> > TestRebalanceLeaders.checkZkLeadersAgree(). The theory
> > is that the identified leader node in ZK is actually the leader
> > after the rebalance command. But you're right, I don't see
> > an actual check that the collection's status agrees.
> >
> > That aside, though, there are several problems I'm uncovering
> >
> > 1> BALANCESHARDUNIQUE can wind up with multiple
> > "preferredLeader" properties defined. Some time between
> > the original code and now someone refactored a bunch of
> > code and missed removing a unique property if it was
> > already assigned and being assigned to another replica
> > in the same slice.
> >
> > 2> to make it much worse, I've rewritten the tests
> > extensively and I can beast the rewritten tests 1,000
> > times and no failures. If I test manually by just issuing
> > the commands, everything works fine. By "testing manually"
> > I mean (working with 4 Vms, 10 shards 4 replicas)
> >> create the collection
> >> issue the BALANCESHARDUNIQUE command
> >> issue the REBALANCELEADERS command
> >
> >
> > However, if instead I
> >> create the collection
> >> issue the BALANCESHARDUNIQUE command
> >> shut down 3 of 4 Solr instances so all the leaders
> >     are on the same host.
> >> restart the 3 instances
> >> issue the REBALANCELEADERS command then
> >     it doesn't work.
> >
> > At least that's what I think I'm seeing, but it makes no
> > real sense yet.
> >
> > So I'm first trying to understand why my manual test
> > fails so regularly, then I can incorporate that setup
> > into the unit test (I'm thinking of just shutting down
> > and restarting some of the Jetty instances).
> >
> > But it's a total mystery to me why restarting Solr instances
> > should have any effect. But that's certainly not
> > something that happens in the current test so I have
> > hopes that tracking that down will lead to understanding
> > what the invalid assumption I'm making is and we can
> > test for that too.,
> >
> > On Tue, Jan 8, 2019 at 1:42 AM Bernd Fehling
> > <[hidden email]> wrote:
> >>
> >> Hi Erick,
> >>
> >> after some more hours of debugging the rough result is, who ever invented
> >> this leader election did not check if an action returns the estimated
> >> result. There are only checks for exceptions, true/false, new sequence
> >> numbers and so on, but never if a leader election to the preferredleader
> >> really took place.
> >>
> >> If doing a rebalanceleaders to preferredleader I also have to check if:
> >> - a rebalance took place
> >> - the preferredleader has really become leader (and not anyone else)
> >>
> >> Currently this is not checked and the call rebalanceleaders to preferredleader
> >> is like a shot into the dark with hope of success. And thats why any
> >> problems have never been discovered or reported.
> >>
> >> Bernd
> >>
> >>
> >> Am 21.12.18 um 18:00 schrieb Erick Erickson:
> >>> I looked at the test last night and it's...disturbing. It succeeds
> >>> 100% of the time. Manual testing seems to fail very often.
> >>> Of course it was late and I was a bit cross-eyed, so maybe
> >>> I wasn't looking at the manual tests correctly. Or maybe the
> >>> test is buggy.
> >>>
> >>> I beasted the test 100x last night and all of them succeeded.
> >>>
> >>> This was with all NRT replicas.
> >>>
> >>> Today I'm going to modify the test into a stand-alone program
> >>> to see if it's something in the test environment that causes
> >>> it to succeed. I've got to get this to fail as a unit test before I
> >>> have confidence in any fixes, and also confidence that things
> >>> like this will be caught going forward.
> >>>
> >>> Erick
> >>>
> >>> On Fri, Dec 21, 2018 at 3:59 AM Bernd Fehling
> >>> <[hidden email]> wrote:
> >>>>
> >>>> As far as I could see with debugger there is still a problem in requeing.
> >>>>
> >>>> There is a watcher and it is recognized that the watcher is not a preferredleader.
> >>>> So it tries to locate a preferredleader with success.
> >>>> It then calls makeReplicaFirstWatcher and gets a new sequence number for
> >>>> the preferredleader replica. But now we have two replicas with the same
> >>>> sequence number. One replica which already owns that sequence number and
> >>>> the replica which got the new (and the same) number as new sequence number.
> >>>> It now tries to solve this with queueNodesWithSameSequence.
> >>>> Might be something in rejoinElection.
> >>>> At least the call to rejoinElection seems right. For preferredleader it
> >>>> is true for rejoinAtHead and for the other replica with same sequence number
> >>>> it is false for rejoinAtHead.
> >>>>
> >>>> A test case should have 3 shards with 3 cores per shard and should try to
> >>>> set preferredleader to different replicas at random. And then try to
> >>>> rebalance and check the results.
> >>>>
> >>>> So far, regards, Bernd
> >>>>
> >>>>
> >>>> Am 21.12.18 um 07:11 schrieb Erick Erickson:
> >>>>> I'm reworking the test case, so hold off on doing that. If you want to
> >>>>> raise a JIRA, though. please do and attach your patch...
> >>>>>
> >>>>> On Thu, Dec 20, 2018 at 10:53 AM Erick Erickson <[hidden email]> wrote:
> >>>>>>
> >>>>>> Nothing that I know of was _intentionally_ changed with this between
> >>>>>> 6x and 7x. That said, nothing that I know of was done to verify that
> >>>>>> TLOG and PULL replicas (added in 7x) were handled correctly. There's a
> >>>>>> test "TestRebalanceLeaders" for this functionality that has run since
> >>>>>> the feature was put in, but it has _not_ been modified to create TLOG
> >>>>>> and PULL replicas and test with those.
> >>>>>>
> >>>>>> For this patch to be complete, we should either extend that test or
> >>>>>> make another that fails without this patch and succeeds with it.
> >>>>>>
> >>>>>> I'd probably recommend modifying TestRebalanceLeaders to randomly
> >>>>>> create TLOG and (maybe) PULL replicas so we'd keep covering the
> >>>>>> various cases.
> >>>>>>
> >>>>>> Best,
> >>>>>> Erick
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Dec 20, 2018 at 8:06 AM Bernd Fehling
> >>>>>> <[hidden email]> wrote:
> >>>>>>>
> >>>>>>> Hi Vadim,
> >>>>>>> I just tried it with 6.6.5.
> >>>>>>> In my test cloud with 5 shards, 5 nodes, 3 cores per node it missed
> >>>>>>> one shard to become leader. But noticed that one shard already was
> >>>>>>> leader. No errors or exceptions in logs.
> >>>>>>> May be I should enable debug logging and try again to see all logging
> >>>>>>> messages from the patch.
> >>>>>>>
> >>>>>>> Might be they also changed other parts between 6.6.5 and 7.6.0 so that
> >>>>>>> it works for you.
> >>>>>>>
> >>>>>>> I also just changed from zookeeper 3.4.10 to 3.4.13 which works fine,
> >>>>>>> even with 3.4.10 dataDir. No errors no complains. Seems to be compatible.
> >>>>>>>
> >>>>>>> Regards, Bernd
> >>>>>>>
> >>>>>>>
> >>>>>>> Am 20.12.18 um 12:31 schrieb Vadim Ivanov:
> >>>>>>>> Yes! It works!
> >>>>>>>> I have tested RebalanceLeaders today with the patch provided by Endika Posadas. (http://lucene.472066.n3.nabble.com/Rebalance-Leaders-Leader-node-deleted-when-rebalancing-leaders-td4417040.html)
> >>>>>>>> And at last it works as expected on my collection with 5 nodes and about 400 shards.
> >>>>>>>> Original patch was slightly incompatible with 7.6.0
> >>>>>>>> I hope this patch will help to try this feature with 7.6
> >>>>>>>> https://drive.google.com/file/d/19z_MPjxItGyghTjXr6zTCVsiSJg1tN20
> >>>>>>>>
> >>>>>>>> RebalanceLeaders was not very useful feature before 7.0 (as all replicas were NRT)
> >>>>>>>> But new replica types made it very helpful to keep big clusters in order...
> >>>>>>>>
> >>>>>>>> I wonder, why there is no any jira about this case (or maybe I missed it)?
> >>>>>>>> Anyone who cares, please, help to create jira and improve this feature in the nearest releaase
> >>>>>>>>
12