A question of solr recovery

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

A question of solr recovery

Hongxu Ma
Hi all
In my cluster, Solr node turned into long time recovery sometimes.
So I want to know more about recovery and have read a good blog:
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

It mentioned in the recovery section:
"Replays the documents from its own tlog if < 100 new updates have been received by the leader. "

My question: what's the meaning of "updates"? commits? or documents?
I refered solr code but still not sure about it.

Hope you can help, thanks.

Reply | Threaded
Open this post in threaded view
|

Re: A question of solr recovery

Erick Erickson
Updates in this context are individual documents, either new ones
or a new version of an existing document. Long recoveries are
quite unlikely to be replaying a few documents from the tlog.

My bet is that you had to do a “full sync” (there should be messages
to that effect in the Solr log). This means that the replica had to
copy the entire index from the leader, and that varies with the size
of the index, network speed and contention, etc.

And to make it more complicated, and despite the comment about 100
docs and the tlog…. while that copy is going on, _new_ updates are
written to the tlog of the recovering replica and after the index
has been copied, those new updates are replayed locally. The 100
doc limit does _not_ apply in this case. So say the recovery starts
at time T and lasts for 60 seconds. All updates sent to the shard
leader over that 60 seconds are put in the local tlog and after the
copy is done, they’re replayed. And then, you guessed it, any
updates received by the leader over that 60 second period are written
to the recovering replica’s tlog and replayed… Under heavy
indexing loads, this can go no for quite a long time. Not certain
that’s what’s happening, but something to be aware of.

Best,
Erick

> On Dec 10, 2019, at 10:39 PM, Hongxu Ma <[hidden email]> wrote:
>
> Hi all
> In my cluster, Solr node turned into long time recovery sometimes.
> So I want to know more about recovery and have read a good blog:
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> It mentioned in the recovery section:
> "Replays the documents from its own tlog if < 100 new updates have been received by the leader. "
>
> My question: what's the meaning of "updates"? commits? or documents?
> I refered solr code but still not sure about it.
>
> Hope you can help, thanks.
>

Reply | Threaded
Open this post in threaded view
|

Re: A question of solr recovery

Hongxu Ma
Thank you very much @Erick Erickson<mailto:[hidden email]>
It's very clear.

And I found my "full sync" log:
"IndexFetcher Total time taken for download (fullCopy=true,bytesDownloaded=178161685180) : 4377 secs (40704063 bytes/sec) to NIOFSDirectory@..."

A more question:
Form the log, looks it downloaded all segment files (178GB), it's very big and took a long time.
Is it possible only download the segment file which contains the missing part? No need all files, maybe it can save time?

For example, there is my fabricated algorithm (like database does):

  *   recovery form local tlog as much as possible
  *   calculate the latest version
  *   only download the segment file which contains data > this version

Thanks.

________________________________
From: Erick Erickson <[hidden email]>
Sent: Wednesday, December 11, 2019 20:56
To: [hidden email] <[hidden email]>
Subject: Re: A question of solr recovery

Updates in this context are individual documents, either new ones
or a new version of an existing document. Long recoveries are
quite unlikely to be replaying a few documents from the tlog.

My bet is that you had to do a “full sync” (there should be messages
to that effect in the Solr log). This means that the replica had to
copy the entire index from the leader, and that varies with the size
of the index, network speed and contention, etc.

And to make it more complicated, and despite the comment about 100
docs and the tlog…. while that copy is going on, _new_ updates are
written to the tlog of the recovering replica and after the index
has been copied, those new updates are replayed locally. The 100
doc limit does _not_ apply in this case. So say the recovery starts
at time T and lasts for 60 seconds. All updates sent to the shard
leader over that 60 seconds are put in the local tlog and after the
copy is done, they’re replayed. And then, you guessed it, any
updates received by the leader over that 60 second period are written
to the recovering replica’s tlog and replayed… Under heavy
indexing loads, this can go no for quite a long time. Not certain
that’s what’s happening, but something to be aware of.

Best,
Erick

> On Dec 10, 2019, at 10:39 PM, Hongxu Ma <[hidden email]> wrote:
>
> Hi all
> In my cluster, Solr node turned into long time recovery sometimes.
> So I want to know more about recovery and have read a good blog:
> https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/
>
> It mentioned in the recovery section:
> "Replays the documents from its own tlog if < 100 new updates have been received by the leader. "
>
> My question: what's the meaning of "updates"? commits? or documents?
> I refered solr code but still not sure about it.
>
> Hope you can help, thanks.
>

Reply | Threaded
Open this post in threaded view
|

Re: A question of solr recovery

Shawn Heisey-2
On 12/12/2019 3:37 AM, Hongxu Ma wrote:
> And I found my "full sync" log:
> "IndexFetcher Total time taken for download (fullCopy=true,bytesDownloaded=178161685180) : 4377 secs (40704063 bytes/sec) to NIOFSDirectory@..."
>
> A more question:
> Form the log, looks it downloaded all segment files (178GB), it's very big and took a long time.
> Is it possible only download the segment file which contains the missing part? No need all files, maybe it can save time?

I'm finding the code in the replication handler to be difficult to
follow ... but I do not think there is any way to have it do a partial
or catchup backup copy.  I believe it must copy the entire index every time.

To achieve what you're after, using an alternate backup strategy would
be advisable.  There are two good ways:

1) Assuming you're on an OS that supports them, make a hardlink copy of
the data directory to an alternate location on the same filesystem.
This should happen almost instantly.  Then at your leisure, copy from
the hardlink copy to your final destination.  When finished, delete the
hardlink copy.

2) Use rsync (probably with "-avH --delete" options) to copy from the
current index directory to another location.  This will skip any
segments that already exist in the destination.

You could potentially even combine those two options in your final solution.

I do not think the replication handler deals with tlog files at all.
The transaction log capability did not exist when the replication
handler was built.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: A question of solr recovery

Shawn Heisey-2
On 12/12/2019 8:53 AM, Shawn Heisey wrote:
> I do not think the replication handler deals with tlog files at all. The
> transaction log capability did not exist when the replication handler
> was built.

I may have mixed up your message with a different one.  Looking back
over this, I don't see any mention of the replication handler ...
that's in another thread about making backups.  Apologies.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: A question of solr recovery

Hongxu Ma
In reply to this post by Hongxu Ma
Thank you @Erick Erickson for your explanation! (although I don't fully understand all details 🙂 ).

I am using Solr 6.6, so I think there is only NRT replica in this version, and I understand the whole recovery process now.

Maybe I will upgrade to Solr 7+ in future, and try the new TLOG/PULL replica.
Thanks.


________________________________
From: Erick Erickson <[hidden email]>
Sent: Thursday, December 12, 2019 22:49
To: Hongxu Ma <[hidden email]>
Subject: Re: A question of solr recovery

If you’re using TLOG/PULL replica types, then only changed segments
are downloaded. That replication pattern has a very different
algorithm. The problem with NRT replicas is that segments on
different replicas may not contain the same documents (in fact,
almost all the time won’t). This is because the wall-clock time
that the autocommit interval expires at, which closes segments,
will be different due to network delays and the like. This was
a deliberate design choice to make indexing as fast as possible
in distributed situations. If the leader coordinated all the commits,
it’d introduce a delay, potentially quite long if, say, the leader
needed to wait for a timeout.

Even if commits were exactly synchronous over all replicas in a shard,
the leader indexes a document and forwards it to the replica. The
commit could expire on both while the doc was in-flight.

Best,
Erick

On Dec 12, 2019, at 5:37 AM, Hongxu Ma <[hidden email]> wrote:

Thank you very much @Erick Erickson
It's very clear.

And I found my "full sync" log:
"IndexFetcher Total time taken for download (fullCopy=true,bytesDownloaded=178161685180) : 4377 secs (40704063 bytes/sec) to NIOFSDirectory@..."

A more question:
Form the log, looks it downloaded all segment files (178GB), it's very big and took a long time.
Is it possible only download the segment file which contains the missing part? No need all files, maybe it can save time?

For example, there is my fabricated algorithm (like database does):
• recovery form local tlog as much as possible
• calculate the latest version
• only download the segment file which contains data > this version
Thanks.

From: Erick Erickson <[hidden email]>
Sent: Wednesday, December 11, 2019 20:56
To: [hidden email] <[hidden email]>
Subject: Re: A question of solr recovery

Updates in this context are individual documents, either new ones
or a new version of an existing document. Long recoveries are
quite unlikely to be replaying a few documents from the tlog.

My bet is that you had to do a “full sync” (there should be messages
to that effect in the Solr log). This means that the replica had to
copy the entire index from the leader, and that varies with the size
of the index, network speed and contention, etc.

And to make it more complicated, and despite the comment about 100
docs and the tlog…. while that copy is going on, _new_ updates are
written to the tlog of the recovering replica and after the index
has been copied, those new updates are replayed locally. The 100
doc limit does _not_ apply in this case. So say the recovery starts
at time T and lasts for 60 seconds. All updates sent to the shard
leader over that 60 seconds are put in the local tlog and after the
copy is done, they’re replayed. And then, you guessed it, any
updates received by the leader over that 60 second period are written
to the recovering replica’s tlog and replayed… Under heavy
indexing loads, this can go no for quite a long time. Not certain
that’s what’s happening, but something to be aware of.

Best,
Erick

On Dec 10, 2019, at 10:39 PM, Hongxu Ma <[hidden email]> wrote:

Hi all
In my cluster, Solr node turned into long time recovery sometimes.
So I want to know more about recovery and have read a good blog:
https://lucidworks.com/post/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/

It mentioned in the recovery section:
"Replays the documents from its own tlog if < 100 new updates have been received by the leader. "

My question: what's the meaning of "updates"? commits? or documents?
I refered solr code but still not sure about it.

Hope you can help, thanks.