Question about startup memory usage

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Question about startup memory usage

Hongxu Ma
Hi
I have a solr-cloud cluster with a big collection, after startup (no any search/index operations), its jvm memory usage is 9GB (via top: RES).

Cluster and collection info:
each host: total 64G mem, two solr nodes with -xmx=15G
collection: total 9B billion docs (but each doc is very small: only some bytes), total size 3TB.

My question is:
Is the 9G mem usage after startup normal? If so, I am worried that the follow up index/search operations will cause an OOM error.
And how can I reduce the memory usage? Maybe I should introduce more host with nodes, but besides this, is there any other solution?

Thanks.




Reply | Threaded
Open this post in threaded view
|

Re: Question about startup memory usage

Shawn Heisey-2
On 11/13/2019 2:03 AM, Hongxu Ma wrote:
> I have a solr-cloud cluster with a big collection, after startup (no any search/index operations), its jvm memory usage is 9GB (via top: RES).
>
> Cluster and collection info:
> each host: total 64G mem, two solr nodes with -xmx=15G
> collection: total 9B billion docs (but each doc is very small: only some bytes), total size 3TB.
>
> My question is:
> Is the 9G mem usage after startup normal? If so, I am worried that the follow up index/search operations will cause an OOM error.
> And how can I reduce the memory usage? Maybe I should introduce more host with nodes, but besides this, is there any other solution?

With the "-Xmx=15G" option, you've told Java that it can use up to 15GB
for heap.  It's total resident memory usage is eventually going to reach
a little over 15GB and probably never go down.  This is how Java works.

The amount of memory that Java allocates immediately on program startup
is related to the -Xms setting.  Normally Solr uses the same number for
both -Xms and -Xmx, but that can be changed if you desire.  We recommend
using the same number.  If -Xms is smaller than -Xmx, Java may allocate
less memory as soon as it starts, then Solr is going to run through its
startup procedure.  We will not know exactly how much memory allocation
is going to occur when that happens ... but with billions of documents,
it's not going to be small.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Question about startup memory usage

Hongxu Ma
Thank you @Shawn Heisey<mailto:[hidden email]> , you help me many times.

My -xms=1G
When restart solr, I can see the progress of memory increasing (from 1G to 9G, took near 10s).

I have a guess: maybe solr is loading some needed files into heap memory, e.g. *.tip : term index file. What's your thoughts?

thanks.


________________________________
From: Shawn Heisey <[hidden email]>
Sent: Thursday, November 14, 2019 1:15
To: [hidden email] <[hidden email]>
Subject: Re: Question about startup memory usage

On 11/13/2019 2:03 AM, Hongxu Ma wrote:
> I have a solr-cloud cluster with a big collection, after startup (no any search/index operations), its jvm memory usage is 9GB (via top: RES).
>
> Cluster and collection info:
> each host: total 64G mem, two solr nodes with -xmx=15G
> collection: total 9B billion docs (but each doc is very small: only some bytes), total size 3TB.
>
> My question is:
> Is the 9G mem usage after startup normal? If so, I am worried that the follow up index/search operations will cause an OOM error.
> And how can I reduce the memory usage? Maybe I should introduce more host with nodes, but besides this, is there any other solution?

With the "-Xmx=15G" option, you've told Java that it can use up to 15GB
for heap.  It's total resident memory usage is eventually going to reach
a little over 15GB and probably never go down.  This is how Java works.

The amount of memory that Java allocates immediately on program startup
is related to the -Xms setting.  Normally Solr uses the same number for
both -Xms and -Xmx, but that can be changed if you desire.  We recommend
using the same number.  If -Xms is smaller than -Xmx, Java may allocate
less memory as soon as it starts, then Solr is going to run through its
startup procedure.  We will not know exactly how much memory allocation
is going to occur when that happens ... but with billions of documents,
it's not going to be small.

Thanks,
Shawn
Reply | Threaded
Open this post in threaded view
|

Re: Question about startup memory usage

Shawn Heisey-2
On 11/14/2019 1:46 AM, Hongxu Ma wrote:
> Thank you @Shawn Heisey<mailto:[hidden email]> , you help me many times.
>
> My -xms=1G
> When restart solr, I can see the progress of memory increasing (from 1G to 9G, took near 10s).
>
> I have a guess: maybe solr is loading some needed files into heap memory, e.g. *.tip : term index file. What's your thoughts?

Solr's basic operation involves quite a lot of Java memory allocation.
Most of what gets allocated turns into garbage almost immediately, but
Java does not reuse that memory right away ... it can only be reused
after garbage collection on the appropriate memory region runs.

The algorithms in Java that decide between either grabbing more memory
(up to the configured heap limit) or running garbage collection are
beyond my understanding.  For programs with heavy memory allocation,
like Solr, the preference does seem to lean towards allocating more
memory if it's available than performing garbage collection.

I can imagine that initial loading of indexes containing billions of
documents will require quite a bit of heap.  I do not know what data is
stored in that memory.

Thanks,
Shawn