_version_ taking too much memory

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

_version_ taking too much memory

sanjay dutt
Solr Collection setup
Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too much)Heap Space :- 12GB
So basically, above collection is having OutOfMemory issues. And upon inspection, I got to know that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field "_version_" is taking around 50 - 60% memory(5GB).
Now I am trying to understand How "_version_" internally using the field cache?
Around the same time, In Splunk logs I can see "Found MAX value from Terms for _version_ in index". and also few "firstSearcher" queries. Now I think upon core initialization "firstSearcher" triggered which then try to find the Max value for version using Field Cache but because of too many records it occupy lot of memory to carry out the task.
Can someone please validate Is what I am talking in above para is the problem or problem lies in somewhere else?

Best Regards,Sanjay
Reply | Threaded
Open this post in threaded view
|

Re: _version_ taking too much memory

Shalin Shekhar Mangar
You should change the _version_ field to be of type docValues and re-index
data. This way you don't have to pay the memory cost of un-inversion.

On Wed, Nov 18, 2020 at 9:51 AM sanjay dutt
<[hidden email]> wrote:

> Solr Collection setup
> Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too
> much)Heap Space :- 12GB
> So basically, above collection is having OutOfMemory issues. And upon
> inspection, I got to know
> that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field
> "_version_" is taking around 50 - 60% memory(5GB).
> Now I am trying to understand How "_version_" internally using the field
> cache?
> Around the same time, In Splunk logs I can see "Found MAX value from Terms
> for _version_ in index". and also few "firstSearcher" queries. Now I think
> upon core initialization "firstSearcher" triggered which then try to find
> the Max value for version using Field Cache but because of too many records
> it occupy lot of memory to carry out the task.
> Can someone please validate Is what I am talking in above para is the
> problem or problem lies in somewhere else?
>
> Best Regards,Sanjay
>


--
Regards,
Shalin Shekhar Mangar.
Reply | Threaded
Open this post in threaded view
|

Re: _version_ taking too much memory

sanjay dutt
Thank you so much Shalin. Yes that would reduce the memory requirement at run time.But my next question, Is the "firstSearcher" doing this I mean due to which it tries to find the "Max value of version" and for that it it has to do the un-inversion. 
Thanks and Regards,Sanjay Dutt

    On Wednesday, November 18, 2020, 12:54:39 PM GMT+5:30, Shalin Shekhar Mangar <[hidden email]> wrote:  
 
 You should change the _version_ field to be of type docValues and re-index
data. This way you don't have to pay the memory cost of un-inversion.

On Wed, Nov 18, 2020 at 9:51 AM sanjay dutt
<[hidden email]> wrote:

> Solr Collection setup
> Shards :- 2Replication :- 4Documents :- 569 Million (I know it's too
> much)Heap Space :- 12GB
> So basically, above collection is having OutOfMemory issues. And upon
> inspection, I got to know
> that org.apache.lucene.uninverting.FieldCacheImpl$LongsFromArray for field
> "_version_" is taking around 50 - 60% memory(5GB).
> Now I am trying to understand How "_version_" internally using the field
> cache?
> Around the same time, In Splunk logs I can see "Found MAX value from Terms
> for _version_ in index". and also few "firstSearcher" queries. Now I think
> upon core initialization "firstSearcher" triggered which then try to find
> the Max value for version using Field Cache but because of too many records
> it occupy lot of memory to carry out the task.
> Can someone please validate Is what I am talking in above para is the
> problem or problem lies in somewhere else?
>
> Best Regards,Sanjay
>


--
Regards,
Shalin Shekhar Mangar.