Does ConcurrentMergeScheduler actually do smaller merges first?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Does ConcurrentMergeScheduler actually do smaller merges first?

Shawn Heisey-2
Before I open an issue, I would like to double-check my sanity, see if
an issue is needed.

I have noticed that the javadoc for ConcurrentMergeScheduler says that
it schedules smaller merges before larger merges.  In the past, I have
seen evidence suggesting this is not actually the case, that it prefers
larger merges first.

---- background ----

When importing millions of rows from a database using Solr's dataimport
handler, the index will be merged quite frequently while that indexing
occurs.  Eventually, it reaches a point where there are multiple merges
scheduled simultaneously, so the the ongoing indexing thread will be
paused until the number of merges drops below maxMergeCount.

If the smallest merge was being done first, then I don't think the
observed behavior would be what happens.  What I would see happen in the
past is that when a large merge gets scheduled, indexing is paused long
enough for the database connection to time out and be disconnected, so
when the import tries to resume indexing, it can't -- the source
database connection is gone.  For MySQL databases, this timeout takes
about ten minutes to happen. If the smallest merge had completed first,
the count would have decreased long before the database connection could
time out, and indexing would have resumed with no problems.

---- end background ----

The way that I have fixed this problem in the past is to increase
maxMergeCount to 6.  When that's done, the incoming thread never gets
paused, and the database connection doesn't time out.

I can see that the default for maxMergeCount was changed from 2 to 6 in
2014 by LUCENE-6119.  So 5.0 and later probably might not have the
problems I encountered as long as the scheduler is left at defaults ...
but I suspect that the running order of merges goes larger to smaller,
contrary to javadoc.  The code is pretty dense and I haven't completely
deciphered it yet.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Does ConcurrentMergeScheduler actually do smaller merges first?

Michael Sokolov-4
If maxMergeCount was 2, you could get into a situation with three large merges I think; the largest would be paused, but the others could still take > 10 mins to complete. Are you sure that your observation is at odds with what the document says the scheduler is doing?

On Wed, Oct 10, 2018 at 2:28 AM Shawn Heisey <[hidden email]> wrote:
Before I open an issue, I would like to double-check my sanity, see if
an issue is needed.

I have noticed that the javadoc for ConcurrentMergeScheduler says that
it schedules smaller merges before larger merges.  In the past, I have
seen evidence suggesting this is not actually the case, that it prefers
larger merges first.

---- background ----

When importing millions of rows from a database using Solr's dataimport
handler, the index will be merged quite frequently while that indexing
occurs.  Eventually, it reaches a point where there are multiple merges
scheduled simultaneously, so the the ongoing indexing thread will be
paused until the number of merges drops below maxMergeCount.

If the smallest merge was being done first, then I don't think the
observed behavior would be what happens.  What I would see happen in the
past is that when a large merge gets scheduled, indexing is paused long
enough for the database connection to time out and be disconnected, so
when the import tries to resume indexing, it can't -- the source
database connection is gone.  For MySQL databases, this timeout takes
about ten minutes to happen. If the smallest merge had completed first,
the count would have decreased long before the database connection could
time out, and indexing would have resumed with no problems.

---- end background ----

The way that I have fixed this problem in the past is to increase
maxMergeCount to 6.  When that's done, the incoming thread never gets
paused, and the database connection doesn't time out.

I can see that the default for maxMergeCount was changed from 2 to 6 in
2014 by LUCENE-6119.  So 5.0 and later probably might not have the
problems I encountered as long as the scheduler is left at defaults ...
but I suspect that the running order of merges goes larger to smaller,
contrary to javadoc.  The code is pretty dense and I haven't completely
deciphered it yet.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Does ConcurrentMergeScheduler actually do smaller merges first?

Shawn Heisey-2
On 10/10/2018 11:52 AM, Michael Sokolov wrote:
> If maxMergeCount was 2, you could get into a situation with three
> large merges I think; the largest would be paused, but the others
> could still take > 10 mins to complete. Are you sure that your
> observation is at odds with what the document says the scheduler is doing?

I haven't done extremely comprehensive checking, and it has been a
number of years now.  When I was looking, what appeared to be happening
was three merges scheduled.  The smallest one I would expect to complete
in seconds, or certainly within a few minutes. The largest one was
probably at the merge policy's 5GB max segment size, and a merge of that
size would definitely take longer than ten minutes.  I no longer have
access to those indexes, so I can't investigate directly.

There are still new reports on solr-user of database connections failing
while importing millions of rows, even recently.  I have NOT heard about
anyone applying my fix (set maxMergeCount to 6) and still seeing
failures, but I suppose that might have happened.

It is the recent reports of the problem that has prompted me to
investigate deeper and start this thread.  I believe that the merge
scheduler SHOULD handle smaller merges first, just like the javadocs
indicate, but I have seen evidence (at least in the past) that it's not
actually doing so.  My look at the code today seems to indicate that it
is sorting large merges first, but somebody who's intimately familiar
with that code could decipher it a lot faster than I can.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Does ConcurrentMergeScheduler actually do smaller merges first?

Shawn Heisey-2
On 10/10/2018 5:40 PM, Shawn Heisey wrote:
> somebody who's intimately familiar with that code could decipher it a
> lot faster than I can.

I went ahead and built a test class mirroring the sorting code I see in
ConcurrentMergeScheduler (master branch), and it looks like current code
does indeed behave as advertised.  Here's the code I built (paste has a
one month expiration time):

https://apaste.info/CFJT

The output of that class is this, exactly what I was hoping to see:

Index: 0, Size: 4725, Pause: true
Index: 1, Size: 3725, Pause: true
Index: 2, Size: 2725, Pause: true
Index: 3, Size: 1725, Pause: true
Index: 4, Size: 725, Pause: true
Index: 5, Size: 525, Pause: false
Index: 6, Size: 25, Pause: false

The code could use more comments documenting its operation, but it does
look like it's correct, at least in the master branch.

Looking over the commit history for the file, nothing jumped out at me
as being a change that might have reversed the sort order, but I can say
that Solr 1.4.x (Lucene 2.9) is where I saw the problem, and I'm
reasonably certain that some of the reports I handled on the mailing
list were on version 4.x.  I cannot confirm the version on more recent
reports without checking list history.

Thanks,
Shawn


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]