Index size variation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Index size variation

Chris Bamford
Hello

I have observed that sometimes my index size temporarily increases by a large amount, presumably while it it merges segments.

Is there some documentation on this subject? I am trying to estimate total disk space I'll need for a project.

Thanks

Chris




---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

RE: Index size variation

Uwe Schindler
Hi Chris,

as always: "it depends". Generally I would reserve space of approximately the "original" index size. Most indexes that are continuously updated have an overhead by 40% for deleted documents. The remaining 60% are indeed used for merging. Please also keep in mind that if IndexReaders are open while merging, you may see not all used disk space from the directory listing, because some files that were merged but are still open in IndexReaders already disappeared from directory listing (deleted during merge), but still consume disk space (delete on last close POSIX semantics).

If you also "forceMerge" your index (you should never ever do this), you may temporarily need 3 times the index size.

Uwe

-----
Uwe Schindler
Achterdiek 19, D-28357 Bremen
http://www.thetaphi.de
eMail: [hidden email]

> -----Original Message-----
> From: Chris Bamford [mailto:[hidden email]]
> Sent: Friday, March 3, 2017 7:24 PM
> To: [hidden email]
> Subject: Index size variation
>
> Hello
>
> I have observed that sometimes my index size temporarily increases by a
> large amount, presumably while it it merges segments.
>
> Is there some documentation on this subject? I am trying to estimate total
> disk space I'll need for a project.
>
> Thanks
>
> Chris
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Index size variation

Chris Bamford-5
Thanks Uwe, very useful indeed.

Chris


> On 3 Mar 2017, at 18:37, Uwe Schindler <[hidden email]> wrote:
>
> Hi Chris,
>
> as always: "it depends". Generally I would reserve space of approximately the "original" index size. Most indexes that are continuously updated have an overhead by 40% for deleted documents. The remaining 60% are indeed used for merging. Please also keep in mind that if IndexReaders are open while merging, you may see not all used disk space from the directory listing, because some files that were merged but are still open in IndexReaders already disappeared from directory listing (deleted during merge), but still consume disk space (delete on last close POSIX semantics).
>
> If you also "forceMerge" your index (you should never ever do this), you may temporarily need 3 times the index size.
>
> Uwe
>
> -----
> Uwe Schindler
> Achterdiek 19, D-28357 Bremen
> http://www.thetaphi.de
> eMail: [hidden email]
>
>> -----Original Message-----
>> From: Chris Bamford [mailto:[hidden email]]
>> Sent: Friday, March 3, 2017 7:24 PM
>> To: [hidden email]
>> Subject: Index size variation
>>
>> Hello
>>
>> I have observed that sometimes my index size temporarily increases by a
>> large amount, presumably while it it merges segments.
>>
>> Is there some documentation on this subject? I am trying to estimate total
>> disk space I'll need for a project.
>>
>> Thanks
>>
>> Chris
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]