Does CDCR Bootstrap sync leaves replica's out of sync

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Does CDCR Bootstrap sync leaves replica's out of sync

Susheel Kumar-3
Does anybody know about known issue where CDCR bootstrap sync leaves the
replica's on target cluster non touched/out of sync.

After I stopped and restart CDCR, it builds my target leaders index but
replica's on target cluster still showing old index / not modified.


Thnx
Reply | Threaded
Open this post in threaded view
|

Re: Does CDCR Bootstrap sync leaves replica's out of sync

Amrit Sarkar
Hi Susheel,

Pretty sure you are talking about this:
https://issues.apache.org/jira/browse/SOLR-11724

Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2
Medium: https://medium.com/@sarkaramrit2

On Mon, Apr 16, 2018 at 11:35 PM, Susheel Kumar <[hidden email]>
wrote:

> Does anybody know about known issue where CDCR bootstrap sync leaves the
> replica's on target cluster non touched/out of sync.
>
> After I stopped and restart CDCR, it builds my target leaders index but
> replica's on target cluster still showing old index / not modified.
>
>
> Thnx
>
Reply | Threaded
Open this post in threaded view
|

Re: Does CDCR Bootstrap sync leaves replica's out of sync

Tom Peters
There are two ways I've gotten around this issue:

1. Add replicas in the target data center after CDCR bootstrapping has completed.

-or-

2. After the bootstrapping has completed, restart the replica nodes one-at-time in the target data center (restart, wait for replica to catch up, then restart the next).


I recommend doing method #1 over #2 if you can. If you accidentally restart the leader node using method #2, it will promote an out-of-sync replica to the leader and all followers will receive that out-of-date index.

I also recommend pausing indexing if you can while you let the target replicas catch up. I have run into issues where the replicas will not catch up if the leader has a fair amount of updates to replay from the source.

> On Apr 16, 2018, at 2:15 PM, Amrit Sarkar <[hidden email]> wrote:
>
> Hi Susheel,
>
> Pretty sure you are talking about this:
> https://issues.apache.org/jira/browse/SOLR-11724
>
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> Medium: https://medium.com/@sarkaramrit2
>
> On Mon, Apr 16, 2018 at 11:35 PM, Susheel Kumar <[hidden email]>
> wrote:
>
>> Does anybody know about known issue where CDCR bootstrap sync leaves the
>> replica's on target cluster non touched/out of sync.
>>
>> After I stopped and restart CDCR, it builds my target leaders index but
>> replica's on target cluster still showing old index / not modified.
>>
>>
>> Thnx
>>



This message and any attachment may contain information that is confidential and/or proprietary. Any use, disclosure, copying, storing, or distribution of this e-mail or any attached file by anyone other than the intended recipient is strictly prohibited. If you have received this message in error, please notify the sender by reply email and delete the message and any attachments. Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: Does CDCR Bootstrap sync leaves replica's out of sync

Susheel Kumar-3
Thanks Amrit, Peter. I'll go with option#2 but what else i am seeing is
that after bootstrap, target has not been synched further (even though we
have continous indexing happening in source) which I believe due to  the
leaders on source cluster shows updateLogSynchronizer stopped while
replica's on source cluster shows updateLogSynchronizer started.

How can we start updateLogSynchronizer on leader and stop updateLogSynchronizer
on replica on source without switching the leader/follower. Any idea?

Thnx



On Mon, Apr 16, 2018 at 2:20 PM, Tom Peters <[hidden email]> wrote:

> There are two ways I've gotten around this issue:
>
> 1. Add replicas in the target data center after CDCR bootstrapping has
> completed.
>
> -or-
>
> 2. After the bootstrapping has completed, restart the replica nodes
> one-at-time in the target data center (restart, wait for replica to catch
> up, then restart the next).
>
>
> I recommend doing method #1 over #2 if you can. If you accidentally
> restart the leader node using method #2, it will promote an out-of-sync
> replica to the leader and all followers will receive that out-of-date index.
>
> I also recommend pausing indexing if you can while you let the target
> replicas catch up. I have run into issues where the replicas will not catch
> up if the leader has a fair amount of updates to replay from the source.
>
> > On Apr 16, 2018, at 2:15 PM, Amrit Sarkar <[hidden email]>
> wrote:
> >
> > Hi Susheel,
> >
> > Pretty sure you are talking about this:
> > https://issues.apache.org/jira/browse/SOLR-11724
> >
> > Amrit Sarkar
> > Search Engineer
> > Lucidworks, Inc.
> > 415-589-9269
> > www.lucidworks.com
> > Twitter http://twitter.com/lucidworks
> > LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> > Medium: https://medium.com/@sarkaramrit2
> >
> > On Mon, Apr 16, 2018 at 11:35 PM, Susheel Kumar <[hidden email]>
> > wrote:
> >
> >> Does anybody know about known issue where CDCR bootstrap sync leaves the
> >> replica's on target cluster non touched/out of sync.
> >>
> >> After I stopped and restart CDCR, it builds my target leaders index but
> >> replica's on target cluster still showing old index / not modified.
> >>
> >>
> >> Thnx
> >>
>
>
>
> This message and any attachment may contain information that is
> confidential and/or proprietary. Any use, disclosure, copying, storing, or
> distribution of this e-mail or any attached file by anyone other than the
> intended recipient is strictly prohibited. If you have received this
> message in error, please notify the sender by reply email and delete the
> message and any attachments. Thank you.
>