Can we change forceMerge to not need as much disk space?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Can we change forceMerge to not need as much disk space?

Erick Erickson
It’s always bothered me that optimize/forceMerge needs 100% of the disk space. I’ve recently been wondering whether that’s absolutely necessary, especially now that forceMerge respects the max segment size.

I HAVE NOT looked at the code closely, so this is mostly theory for someone to shoot down before diving in at all.

I’ve seen some situations where optimizing makes a radical difference. For instance, the time it takes for the Terms component to return is essentially linear to the number of segments. An artificially bad case to be sure, but still. We’re talking the difference between 17 seconds and sub-second here. A large index to be sure…

Anyway, it occurred to me that once a max-sized segment is created, _if_ we write the segments_n file out with the current state of the index, we could freely delete the segments that were merged into the new one. With 300G indexes (which I see regularly in the field, even multiple ones per node that size), this could result in substantial disk savings.

Off the top of my head, I can see some concerns:
1> we’d have to open new searchers every time we wrote the segments_n file to release file handles on the old segments
2> coordinating multiple merge threads
3> maxMergeAtOnceExplicit could mean unnecessary thrashing/opening searchers (could this be deprecated?)
4> Don’t quite know what to do if maxSegments is 1 (or other very low number).

Something like this would also pave the way for “background optimizing”. Instead of a monolithic forceMerge, I can envision a process whereby we created a low-level task that merged one max-sized segment at a time, came up for air and reopened searchers then went back in and merged the next one. With its own problems about coordinating ongoing updates, but that’s another discussion ;).

There’s lots of details to work out, throwing this out for discussion. I can raise a JIRA if people think the idea has legs.

Erick



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Can we change forceMerge to not need as much disk space?

Bram Van Dam
On 02/09/2019 17:19, Erick Erickson wrote:
> 4> Don’t quite know what to do if maxSegments is 1 (or other very low number).

Having maxSegments set to > 5 (or whatever) seems like an acceptable
constraint if it enables optimize without 200% disk usage.

> Something like this would also pave the way for “background optimizing”. Instead of a monolithic forceMerge, I can envision a process whereby we created a low-level task that merged one max-sized segment at a time, came up for air and reopened searchers then went back in and merged the next one. With its own problems about coordinating ongoing updates, but that’s another discussion ;).
>
> There’s lots of details to work out, throwing this out for discussion. I can raise a JIRA if people think the idea has legs.

Without having looked at the code, and going only on your assumptions
and my own observations: it sounds like a good idea. The idea of a
background optimizing process is particularly tantalizing.

AFAICT there hasn't been any other feedback re this? :-/

 - Bram

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Can we change forceMerge to not need as much disk space?

Michael McCandless-2
I think this is worth exploring?

Essentially, after each large merge, we'd need to 1) commit, and 2) refresh any open readers (and close the old readers), to fully free up transient disk usage.  Maybe we could somehow track the current transient extra disk usage of the index + open readers and once that exceeds a threshold, do something.  The "something" could even be asynchronous, e.g. maybe the next merge kicks off, and then asynchronously your app calls commit / refresh?  It could be an event/listener API that IW invokes maybe ...

However, the final merge (if merging to a single segment) will necessarily consume up to 2X the index size (1X for the current index + 1X for the newly merged segment); I don't see how to reduce that requirement for the final merge.

On Fri, Sep 13, 2019 at 12:54 PM Bram Van Dam <[hidden email]> wrote:
On 02/09/2019 17:19, Erick Erickson wrote:
> 4> Don’t quite know what to do if maxSegments is 1 (or other very low number).

Having maxSegments set to > 5 (or whatever) seems like an acceptable
constraint if it enables optimize without 200% disk usage.

> Something like this would also pave the way for “background optimizing”. Instead of a monolithic forceMerge, I can envision a process whereby we created a low-level task that merged one max-sized segment at a time, came up for air and reopened searchers then went back in and merged the next one. With its own problems about coordinating ongoing updates, but that’s another discussion ;).
>
> There’s lots of details to work out, throwing this out for discussion. I can raise a JIRA if people think the idea has legs.

Without having looked at the code, and going only on your assumptions
and my own observations: it sounds like a good idea. The idea of a
background optimizing process is particularly tantalizing.

AFAICT there hasn't been any other feedback re this? :-/

 - Bram
Reply | Threaded
Open this post in threaded view
|

Re: Can we change forceMerge to not need as much disk space?

Erick Erickson
Yeah, if merging down to 1 segment, there’s no choice in the matter. But with the changes to TMP in 7.5, users have to explicitly use maxSegments=1 to get that behavior.

It also seems we could usefully predict how much free space we need, each merge thread does have a list of segments, can estimate the disk space needed based on the pct deleted docs in each segment etc.

I can imagine aborting the merge if disk space is at some threshold free space too. Off the top of my head, I can imagine each merge thread trying to predict how much space it needs and updating some global var with that number, that way other cores in the same JVM would have something to check to prevent race conditions.

But even if something like that worked to perfection, it’s still possible to have N JVMs running at the same time going at the same disk, so the problem would still be subject to race conditions. I suppose the merge process could periodically check if there was still enough free space to succeed (plus some slop of course).

A lot of this is noodling. Right now I’m traveling and have some free time…  I’m not ready to spend a lot of time on soon.

> On Sep 13, 2019, at 2:34 PM, Michael McCandless <[hidden email]> wrote:
>
> I think this is worth exploring?
>
> Essentially, after each large merge, we'd need to 1) commit, and 2) refresh any open readers (and close the old readers), to fully free up transient disk usage.  Maybe we could somehow track the current transient extra disk usage of the index + open readers and once that exceeds a threshold, do something.  The "something" could even be asynchronous, e.g. maybe the next merge kicks off, and then asynchronously your app calls commit / refresh?  It could be an event/listener API that IW invokes maybe ...
>
> However, the final merge (if merging to a single segment) will necessarily consume up to 2X the index size (1X for the current index + 1X for the newly merged segment); I don't see how to reduce that requirement for the final merge.
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Fri, Sep 13, 2019 at 12:54 PM Bram Van Dam <[hidden email]> wrote:
> On 02/09/2019 17:19, Erick Erickson wrote:
> > 4> Don’t quite know what to do if maxSegments is 1 (or other very low number).
>
> Having maxSegments set to > 5 (or whatever) seems like an acceptable
> constraint if it enables optimize without 200% disk usage.
>
> > Something like this would also pave the way for “background optimizing”. Instead of a monolithic forceMerge, I can envision a process whereby we created a low-level task that merged one max-sized segment at a time, came up for air and reopened searchers then went back in and merged the next one. With its own problems about coordinating ongoing updates, but that’s another discussion ;).
> >
> > There’s lots of details to work out, throwing this out for discussion. I can raise a JIRA if people think the idea has legs.
>
> Without having looked at the code, and going only on your assumptions
> and my own observations: it sounds like a good idea. The idea of a
> background optimizing process is particularly tantalizing.
>
> AFAICT there hasn't been any other feedback re this? :-/
>
>  - Bram


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Can we change forceMerge to not need as much disk space?

Shawn Heisey-2
In reply to this post by Erick Erickson
On 9/2/2019 9:19 AM, Erick Erickson wrote:
> Anyway, it occurred to me that once a max-sized segment is created, _if_ we write the segments_n file out with the current state of the index, we could freely delete the segments that were merged into the new one. With 300G indexes (which I see regularly in the field, even multiple ones per node that size), this could result in substantial disk savings.

<snip>

> Off the top of my head, I can see some concerns:
> 1> we’d have to open new searchers every time we wrote the segments_n file to release file handles on the old segments

How would that interact with user applications that normally handle
opening new searchers (such as Solr)?  When users want there to be no
new searchers until they issue an explicit commit, I think they're going
to be a little irritated if Lucene decides to open a new searcher on its
own.  Maybe we'd need to advise people to turn off their indexing
anytime they're doing a forceMerge/optimize.  That's generally a good
idea anyway, and pretty much required if deleteByQuery is being used.

> 2> coordinating multiple merge threads

I would think the scheduler already handles that ... thinking about all
this makes my brain hurt ... if I have to think about the scheduler too,
there might be implosions. :)

> 3> maxMergeAtOnceExplicit could mean unnecessary thrashing/opening searchers (could this be deprecated?)

It has always bothered me that when I looked for info about changing the
policy settings, and set the two "main" parts of the policy to 35
(instead of the default 10), that the info I was finding never mentioned
maxMergeAtOnceExplicit.  I also needed to set this value (to 105) to
have an optimize work like I expected.  Without it, a lot more merging
occurred than was necessary when I did an optimize.  This was on a
really old version of Solr, either 1.4.x or 3.2.x, back when it was
relatively new.

The maxMergeAtOnceExplicit setting is not even mentioned in the Solr ref
guide page about IndexConfig.  I got the information for that setting
from solr-user, when I asked why an optimize with values increased from
10 to 35 was doing more merge passes than I thought it needed.  I think
that either that parameter needs to go away or docs need improvement.

> 4> Don’t quite know what to do if maxSegments is 1 (or other very low number).

I don't think anything can be done about disk usage for that.  Just the
nature of the beast.

> Something like this would also pave the way for “background optimizing”. Instead of a monolithic forceMerge, I can envision a process whereby we created a low-level task that merged one max-sized segment at a time, came up for air and reopened searchers then went back in and merged the next one. With its own problems about coordinating ongoing updates, but that’s another discussion ;).

As mentioned above, I worry about low-level code opening new searchers
because lots of users want to have that be completely under their
control.  Maybe TMP needs another setting to tell it whether or not it's
allowed to open searchers, with documentation saying that less disk
space might be required if it is allowed.

It would be awesome to eliminate the huge forceMerge disk requirement
for most users, so I think it's worth exploring.  Can the stuff with
readers that Mike mentioned happen without opening a new searcher at the
app level?  My knowledge of Lucene internals is unfortunately too vague
to answer my own question.

Thanks,
Shawn

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]