Solr 7.7 heap space is getting full

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Solr 7.7 heap space is getting full

Rajdeep Sahoo
We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb. We
have completed indexing after starting the server suddenly heap space is
getting full.
   Added gc params  , still not working and jdk version is 1.8 .
Please find the below gc  params
-XX:NewRatio=2
-XX:SurvivorRatio=3
-XX:TargetSurvivorRatio=90 \
-XX:MaxTenuringThreshold=8 \
-XX:+UseConcMarkSweepGC \
-XX:+CMSScavengeBeforeRemark \
-XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
-XX:PretenureSizeThreshold=512m \
-XX:CMSFullGCsBeforeCompaction=1 \
-XX:+UseCMSInitiatingOccupancyOnly \
-XX:CMSInitiatingOccupancyFraction=70 \
-XX:CMSMaxAbortablePrecleanTime=6000 \
-XX:+CMSParallelRemarkEnabled
-XX:+ParallelRefProcEnabled
-XX:+UseLargePages \
-XX:+AggressiveOpts \
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Please reply anyone

On Sun, 19 Jan, 2020, 10:55 PM Rajdeep Sahoo, <[hidden email]>
wrote:

> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb. We
> have completed indexing after starting the server suddenly heap space is
> getting full.
>    Added gc params  , still not working and jdk version is 1.8 .
> Please find the below gc  params
> -XX:NewRatio=2
> -XX:SurvivorRatio=3
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
> -XX:+UseConcMarkSweepGC \
> -XX:+CMSScavengeBeforeRemark \
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> -XX:PretenureSizeThreshold=512m \
> -XX:CMSFullGCsBeforeCompaction=1 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=70 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Walter Underwood
In reply to this post by Rajdeep Sahoo
With Java 1.8, I would use the G1 garbage collector. We’ve been running that combination in prod for three years with no problems.

SOLR_HEAP=8g
# Use G1 GC  -- wunder 2017-01-23
# Settings from https://wiki.apache.org/solr/ShawnHeisey
GC_TUNE=" \
-XX:+UseG1GC \
-XX:+ParallelRefProcEnabled \
-XX:G1HeapRegionSize=8m \
-XX:MaxGCPauseMillis=200 \
-XX:+UseLargePages \
-XX:+AggressiveOpts \


wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <[hidden email]> wrote:
>
> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb. We
> have completed indexing after starting the server suddenly heap space is
> getting full.
>   Added gc params  , still not working and jdk version is 1.8 .
> Please find the below gc  params
> -XX:NewRatio=2
> -XX:SurvivorRatio=3
> -XX:TargetSurvivorRatio=90 \
> -XX:MaxTenuringThreshold=8 \
> -XX:+UseConcMarkSweepGC \
> -XX:+CMSScavengeBeforeRemark \
> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> -XX:PretenureSizeThreshold=512m \
> -XX:CMSFullGCsBeforeCompaction=1 \
> -XX:+UseCMSInitiatingOccupancyOnly \
> -XX:CMSInitiatingOccupancyFraction=70 \
> -XX:CMSMaxAbortablePrecleanTime=6000 \
> -XX:+CMSParallelRemarkEnabled
> -XX:+ParallelRefProcEnabled
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Hi,
Still facing the same issue...
Anything else that we need to check.


On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <[hidden email]>
wrote:

> With Java 1.8, I would use the G1 garbage collector. We’ve been running
> that combination in prod for three years with no problems.
>
> SOLR_HEAP=8g
> # Use G1 GC  -- wunder 2017-01-23
> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> GC_TUNE=" \
> -XX:+UseG1GC \
> -XX:+ParallelRefProcEnabled \
> -XX:G1HeapRegionSize=8m \
> -XX:MaxGCPauseMillis=200 \
> -XX:+UseLargePages \
> -XX:+AggressiveOpts \
> “
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <[hidden email]>
> wrote:
> >
> > We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb.
> We
> > have completed indexing after starting the server suddenly heap space is
> > getting full.
> >   Added gc params  , still not working and jdk version is 1.8 .
> > Please find the below gc  params
> > -XX:NewRatio=2
> > -XX:SurvivorRatio=3
> > -XX:TargetSurvivorRatio=90 \
> > -XX:MaxTenuringThreshold=8 \
> > -XX:+UseConcMarkSweepGC \
> > -XX:+CMSScavengeBeforeRemark \
> > -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> > -XX:PretenureSizeThreshold=512m \
> > -XX:CMSFullGCsBeforeCompaction=1 \
> > -XX:+UseCMSInitiatingOccupancyOnly \
> > -XX:CMSInitiatingOccupancyFraction=70 \
> > -XX:CMSMaxAbortablePrecleanTime=6000 \
> > -XX:+CMSParallelRemarkEnabled
> > -XX:+ParallelRefProcEnabled
> > -XX:+UseLargePages \
> > -XX:+AggressiveOpts \
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Walter Underwood
A new garbage collector won’t fix it, but it might help a bit.

Requesting 200 facet fields and having 50-60 of them with results is a huge amount of work for Solr. A typical faceting implementation might have three to five facets. Your requests will be at least 10X to 20X slower.

Check the CPU during one request. It should use nearly 100% of a single CPU. If it a lot lower than 100%, you have another bottleneck. That might be insufficient heap or accessing disk during query requests (not enough RAM). If it is near 100%, the only thing you can do is get a faster CPU.

One other question, how frequently is the index updated?

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <[hidden email]> wrote:
>
> Hi,
> Still facing the same issue...
> Anything else that we need to check.
>
>
> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <[hidden email]>
> wrote:
>
>> With Java 1.8, I would use the G1 garbage collector. We’ve been running
>> that combination in prod for three years with no problems.
>>
>> SOLR_HEAP=8g
>> # Use G1 GC  -- wunder 2017-01-23
>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> GC_TUNE=" \
>> -XX:+UseG1GC \
>> -XX:+ParallelRefProcEnabled \
>> -XX:G1HeapRegionSize=8m \
>> -XX:MaxGCPauseMillis=200 \
>> -XX:+UseLargePages \
>> -XX:+AggressiveOpts \
>> “
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <[hidden email]>
>> wrote:
>>>
>>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb.
>> We
>>> have completed indexing after starting the server suddenly heap space is
>>> getting full.
>>>  Added gc params  , still not working and jdk version is 1.8 .
>>> Please find the below gc  params
>>> -XX:NewRatio=2
>>> -XX:SurvivorRatio=3
>>> -XX:TargetSurvivorRatio=90 \
>>> -XX:MaxTenuringThreshold=8 \
>>> -XX:+UseConcMarkSweepGC \
>>> -XX:+CMSScavengeBeforeRemark \
>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
>>> -XX:PretenureSizeThreshold=512m \
>>> -XX:CMSFullGCsBeforeCompaction=1 \
>>> -XX:+UseCMSInitiatingOccupancyOnly \
>>> -XX:CMSInitiatingOccupancyFraction=70 \
>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
>>> -XX:+CMSParallelRemarkEnabled
>>> -XX:+ParallelRefProcEnabled
>>> -XX:+UseLargePages \
>>> -XX:+AggressiveOpts \
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Hi,
 Currently there is no request or indexing is happening.
  It's just start up
 And during that time heap is getting full.
 Index size is approx 1 g.


On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <[hidden email]>
wrote:

> A new garbage collector won’t fix it, but it might help a bit.
>
> Requesting 200 facet fields and having 50-60 of them with results is a
> huge amount of work for Solr. A typical faceting implementation might have
> three to five facets. Your requests will be at least 10X to 20X slower.
>
> Check the CPU during one request. It should use nearly 100% of a single
> CPU. If it a lot lower than 100%, you have another bottleneck. That might
> be insufficient heap or accessing disk during query requests (not enough
> RAM). If it is near 100%, the only thing you can do is get a faster CPU.
>
> One other question, how frequently is the index updated?
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <[hidden email]>
> wrote:
> >
> > Hi,
> > Still facing the same issue...
> > Anything else that we need to check.
> >
> >
> > On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <[hidden email]>
> > wrote:
> >
> >> With Java 1.8, I would use the G1 garbage collector. We’ve been running
> >> that combination in prod for three years with no problems.
> >>
> >> SOLR_HEAP=8g
> >> # Use G1 GC  -- wunder 2017-01-23
> >> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> >> GC_TUNE=" \
> >> -XX:+UseG1GC \
> >> -XX:+ParallelRefProcEnabled \
> >> -XX:G1HeapRegionSize=8m \
> >> -XX:MaxGCPauseMillis=200 \
> >> -XX:+UseLargePages \
> >> -XX:+AggressiveOpts \
> >> “
> >>
> >> wunder
> >> Walter Underwood
> >> [hidden email]
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <[hidden email]
> >
> >> wrote:
> >>>
> >>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb.
> >> We
> >>> have completed indexing after starting the server suddenly heap space
> is
> >>> getting full.
> >>>  Added gc params  , still not working and jdk version is 1.8 .
> >>> Please find the below gc  params
> >>> -XX:NewRatio=2
> >>> -XX:SurvivorRatio=3
> >>> -XX:TargetSurvivorRatio=90 \
> >>> -XX:MaxTenuringThreshold=8 \
> >>> -XX:+UseConcMarkSweepGC \
> >>> -XX:+CMSScavengeBeforeRemark \
> >>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> >>> -XX:PretenureSizeThreshold=512m \
> >>> -XX:CMSFullGCsBeforeCompaction=1 \
> >>> -XX:+UseCMSInitiatingOccupancyOnly \
> >>> -XX:CMSInitiatingOccupancyFraction=70 \
> >>> -XX:CMSMaxAbortablePrecleanTime=6000 \
> >>> -XX:+CMSParallelRemarkEnabled
> >>> -XX:+ParallelRefProcEnabled
> >>> -XX:+UseLargePages \
> >>> -XX:+AggressiveOpts \
> >>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Walter Underwood
What message do you get that means the heap space is full?

Java will always use all of the heap, either as live data or not-yet-collected garbage.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <[hidden email]> wrote:
>
> Hi,
> Currently there is no request or indexing is happening.
>  It's just start up
> And during that time heap is getting full.
> Index size is approx 1 g.
>
>
> On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <[hidden email]>
> wrote:
>
>> A new garbage collector won’t fix it, but it might help a bit.
>>
>> Requesting 200 facet fields and having 50-60 of them with results is a
>> huge amount of work for Solr. A typical faceting implementation might have
>> three to five facets. Your requests will be at least 10X to 20X slower.
>>
>> Check the CPU during one request. It should use nearly 100% of a single
>> CPU. If it a lot lower than 100%, you have another bottleneck. That might
>> be insufficient heap or accessing disk during query requests (not enough
>> RAM). If it is near 100%, the only thing you can do is get a faster CPU.
>>
>> One other question, how frequently is the index updated?
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <[hidden email]>
>> wrote:
>>>
>>> Hi,
>>> Still facing the same issue...
>>> Anything else that we need to check.
>>>
>>>
>>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <[hidden email]>
>>> wrote:
>>>
>>>> With Java 1.8, I would use the G1 garbage collector. We’ve been running
>>>> that combination in prod for three years with no problems.
>>>>
>>>> SOLR_HEAP=8g
>>>> # Use G1 GC  -- wunder 2017-01-23
>>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>>>> GC_TUNE=" \
>>>> -XX:+UseG1GC \
>>>> -XX:+ParallelRefProcEnabled \
>>>> -XX:G1HeapRegionSize=8m \
>>>> -XX:MaxGCPauseMillis=200 \
>>>> -XX:+UseLargePages \
>>>> -XX:+AggressiveOpts \
>>>> “
>>>>
>>>> wunder
>>>> Walter Underwood
>>>> [hidden email]
>>>> http://observer.wunderwood.org/  (my blog)
>>>>
>>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <[hidden email]
>>>
>>>> wrote:
>>>>>
>>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12 gb.
>>>> We
>>>>> have completed indexing after starting the server suddenly heap space
>> is
>>>>> getting full.
>>>>> Added gc params  , still not working and jdk version is 1.8 .
>>>>> Please find the below gc  params
>>>>> -XX:NewRatio=2
>>>>> -XX:SurvivorRatio=3
>>>>> -XX:TargetSurvivorRatio=90 \
>>>>> -XX:MaxTenuringThreshold=8 \
>>>>> -XX:+UseConcMarkSweepGC \
>>>>> -XX:+CMSScavengeBeforeRemark \
>>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
>>>>> -XX:PretenureSizeThreshold=512m \
>>>>> -XX:CMSFullGCsBeforeCompaction=1 \
>>>>> -XX:+UseCMSInitiatingOccupancyOnly \
>>>>> -XX:CMSInitiatingOccupancyFraction=70 \
>>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
>>>>> -XX:+CMSParallelRemarkEnabled
>>>>> -XX:+ParallelRefProcEnabled
>>>>> -XX:+UseLargePages \
>>>>> -XX:+AggressiveOpts \
>>>>
>>>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Initially we were getting the warning message as  ulimit is low i.e. 1024
so we changed it to 65000
Using ulimit -u 65000.

Then the error was failed to reserve shared memory error =1
 Because of this we removed
   -xx : +uselargepages

Now in console log it is showing
Could not find or load main class \

And solr is not starting up



On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, <[hidden email]>
wrote:

> What message do you get that means the heap space is full?
>
> Java will always use all of the heap, either as live data or
> not-yet-collected garbage.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <[hidden email]>
> wrote:
> >
> > Hi,
> > Currently there is no request or indexing is happening.
> >  It's just start up
> > And during that time heap is getting full.
> > Index size is approx 1 g.
> >
> >
> > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <[hidden email]>
> > wrote:
> >
> >> A new garbage collector won’t fix it, but it might help a bit.
> >>
> >> Requesting 200 facet fields and having 50-60 of them with results is a
> >> huge amount of work for Solr. A typical faceting implementation might
> have
> >> three to five facets. Your requests will be at least 10X to 20X slower.
> >>
> >> Check the CPU during one request. It should use nearly 100% of a single
> >> CPU. If it a lot lower than 100%, you have another bottleneck. That
> might
> >> be insufficient heap or accessing disk during query requests (not enough
> >> RAM). If it is near 100%, the only thing you can do is get a faster CPU.
> >>
> >> One other question, how frequently is the index updated?
> >>
> >> wunder
> >> Walter Underwood
> >> [hidden email]
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <[hidden email]
> >
> >> wrote:
> >>>
> >>> Hi,
> >>> Still facing the same issue...
> >>> Anything else that we need to check.
> >>>
> >>>
> >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <[hidden email]
> >
> >>> wrote:
> >>>
> >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
> running
> >>>> that combination in prod for three years with no problems.
> >>>>
> >>>> SOLR_HEAP=8g
> >>>> # Use G1 GC  -- wunder 2017-01-23
> >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> >>>> GC_TUNE=" \
> >>>> -XX:+UseG1GC \
> >>>> -XX:+ParallelRefProcEnabled \
> >>>> -XX:G1HeapRegionSize=8m \
> >>>> -XX:MaxGCPauseMillis=200 \
> >>>> -XX:+UseLargePages \
> >>>> -XX:+AggressiveOpts \
> >>>> “
> >>>>
> >>>> wunder
> >>>> Walter Underwood
> >>>> [hidden email]
> >>>> http://observer.wunderwood.org/  (my blog)
> >>>>
> >>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <
> [hidden email]
> >>>
> >>>> wrote:
> >>>>>
> >>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12
> gb.
> >>>> We
> >>>>> have completed indexing after starting the server suddenly heap space
> >> is
> >>>>> getting full.
> >>>>> Added gc params  , still not working and jdk version is 1.8 .
> >>>>> Please find the below gc  params
> >>>>> -XX:NewRatio=2
> >>>>> -XX:SurvivorRatio=3
> >>>>> -XX:TargetSurvivorRatio=90 \
> >>>>> -XX:MaxTenuringThreshold=8 \
> >>>>> -XX:+UseConcMarkSweepGC \
> >>>>> -XX:+CMSScavengeBeforeRemark \
> >>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> >>>>> -XX:PretenureSizeThreshold=512m \
> >>>>> -XX:CMSFullGCsBeforeCompaction=1 \
> >>>>> -XX:+UseCMSInitiatingOccupancyOnly \
> >>>>> -XX:CMSInitiatingOccupancyFraction=70 \
> >>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
> >>>>> -XX:+CMSParallelRemarkEnabled
> >>>>> -XX:+ParallelRefProcEnabled
> >>>>> -XX:+UseLargePages \
> >>>>> -XX:+AggressiveOpts \
> >>>>
> >>>>
> >>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Mehai, Lotfi
I  had a similar issue with a large number of facets. There is no way (At
least I know) your can get an acceptable response time from search engine
with high number of facets.
The way we solved the issue was to cache shallow Facets data structure in
the web services. Facts structures are refreshed periodically. We don't
have near real time indexation requirements. Page response time is under
5s.

Here the URLs for our worst use case:
https://www.govinfo.gov/app/collection/cfr
https://www.govinfo.gov/app/cfrparts/month

I hope that helps.

Lotfi Mehai
https://www.linkedin.com/in/lmehai/





On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo <[hidden email]>
wrote:

> Initially we were getting the warning message as  ulimit is low i.e. 1024
> so we changed it to 65000
> Using ulimit -u 65000.
>
> Then the error was failed to reserve shared memory error =1
>  Because of this we removed
>    -xx : +uselargepages
>
> Now in console log it is showing
> Could not find or load main class \
>
> And solr is not starting up
>
>
>
> On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, <[hidden email]>
> wrote:
>
> > What message do you get that means the heap space is full?
> >
> > Java will always use all of the heap, either as live data or
> > not-yet-collected garbage.
> >
> > wunder
> > Walter Underwood
> > [hidden email]
> > http://observer.wunderwood.org/  (my blog)
> >
> > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <[hidden email]
> >
> > wrote:
> > >
> > > Hi,
> > > Currently there is no request or indexing is happening.
> > >  It's just start up
> > > And during that time heap is getting full.
> > > Index size is approx 1 g.
> > >
> > >
> > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <[hidden email]
> >
> > > wrote:
> > >
> > >> A new garbage collector won’t fix it, but it might help a bit.
> > >>
> > >> Requesting 200 facet fields and having 50-60 of them with results is a
> > >> huge amount of work for Solr. A typical faceting implementation might
> > have
> > >> three to five facets. Your requests will be at least 10X to 20X
> slower.
> > >>
> > >> Check the CPU during one request. It should use nearly 100% of a
> single
> > >> CPU. If it a lot lower than 100%, you have another bottleneck. That
> > might
> > >> be insufficient heap or accessing disk during query requests (not
> enough
> > >> RAM). If it is near 100%, the only thing you can do is get a faster
> CPU.
> > >>
> > >> One other question, how frequently is the index updated?
> > >>
> > >> wunder
> > >> Walter Underwood
> > >> [hidden email]
> > >> http://observer.wunderwood.org/  (my blog)
> > >>
> > >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <
> [hidden email]
> > >
> > >> wrote:
> > >>>
> > >>> Hi,
> > >>> Still facing the same issue...
> > >>> Anything else that we need to check.
> > >>>
> > >>>
> > >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <
> [hidden email]
> > >
> > >>> wrote:
> > >>>
> > >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
> > running
> > >>>> that combination in prod for three years with no problems.
> > >>>>
> > >>>> SOLR_HEAP=8g
> > >>>> # Use G1 GC  -- wunder 2017-01-23
> > >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> > >>>> GC_TUNE=" \
> > >>>> -XX:+UseG1GC \
> > >>>> -XX:+ParallelRefProcEnabled \
> > >>>> -XX:G1HeapRegionSize=8m \
> > >>>> -XX:MaxGCPauseMillis=200 \
> > >>>> -XX:+UseLargePages \
> > >>>> -XX:+AggressiveOpts \
> > >>>> “
> > >>>>
> > >>>> wunder
> > >>>> Walter Underwood
> > >>>> [hidden email]
> > >>>> http://observer.wunderwood.org/  (my blog)
> > >>>>
> > >>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <
> > [hidden email]
> > >>>
> > >>>> wrote:
> > >>>>>
> > >>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space is 12
> > gb.
> > >>>> We
> > >>>>> have completed indexing after starting the server suddenly heap
> space
> > >> is
> > >>>>> getting full.
> > >>>>> Added gc params  , still not working and jdk version is 1.8 .
> > >>>>> Please find the below gc  params
> > >>>>> -XX:NewRatio=2
> > >>>>> -XX:SurvivorRatio=3
> > >>>>> -XX:TargetSurvivorRatio=90 \
> > >>>>> -XX:MaxTenuringThreshold=8 \
> > >>>>> -XX:+UseConcMarkSweepGC \
> > >>>>> -XX:+CMSScavengeBeforeRemark \
> > >>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> > >>>>> -XX:PretenureSizeThreshold=512m \
> > >>>>> -XX:CMSFullGCsBeforeCompaction=1 \
> > >>>>> -XX:+UseCMSInitiatingOccupancyOnly \
> > >>>>> -XX:CMSInitiatingOccupancyFraction=70 \
> > >>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
> > >>>>> -XX:+CMSParallelRemarkEnabled
> > >>>>> -XX:+ParallelRefProcEnabled
> > >>>>> -XX:+UseLargePages \
> > >>>>> -XX:+AggressiveOpts \
> > >>>>
> > >>>>
> > >>
> > >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Initially we were getting the warning message as  ulimit is low i.e. 1024
so we changed it to 65000
Using ulimit -u 65000.

Then the error was failed to reserve shared memory error =1
 Because of this we removed
   -xx : +uselargepages

Now in console log it is showing
Could not find or load main class \

And solr is not starting up


On Mon, 20 Jan, 2020, 7:50 AM Mehai, Lotfi, <[hidden email]> wrote:

> I  had a similar issue with a large number of facets. There is no way (At
> least I know) your can get an acceptable response time from search engine
> with high number of facets.
> The way we solved the issue was to cache shallow Facets data structure in
> the web services. Facts structures are refreshed periodically. We don't
> have near real time indexation requirements. Page response time is under
> 5s.
>
> Here the URLs for our worst use case:
> https://www.govinfo.gov/app/collection/cfr
> https://www.govinfo.gov/app/cfrparts/month
>
> I hope that helps.
>
> Lotfi Mehai
> https://www.linkedin.com/in/lmehai/
>
>
>
>
>
> On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo <[hidden email]>
> wrote:
>
> > Initially we were getting the warning message as  ulimit is low i.e. 1024
> > so we changed it to 65000
> > Using ulimit -u 65000.
> >
> > Then the error was failed to reserve shared memory error =1
> >  Because of this we removed
> >    -xx : +uselargepages
> >
> > Now in console log it is showing
> > Could not find or load main class \
> >
> > And solr is not starting up
> >
> >
> >
> > On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, <[hidden email]>
> > wrote:
> >
> > > What message do you get that means the heap space is full?
> > >
> > > Java will always use all of the heap, either as live data or
> > > not-yet-collected garbage.
> > >
> > > wunder
> > > Walter Underwood
> > > [hidden email]
> > > http://observer.wunderwood.org/  (my blog)
> > >
> > > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <
> [hidden email]
> > >
> > > wrote:
> > > >
> > > > Hi,
> > > > Currently there is no request or indexing is happening.
> > > >  It's just start up
> > > > And during that time heap is getting full.
> > > > Index size is approx 1 g.
> > > >
> > > >
> > > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <
> [hidden email]
> > >
> > > > wrote:
> > > >
> > > >> A new garbage collector won’t fix it, but it might help a bit.
> > > >>
> > > >> Requesting 200 facet fields and having 50-60 of them with results
> is a
> > > >> huge amount of work for Solr. A typical faceting implementation
> might
> > > have
> > > >> three to five facets. Your requests will be at least 10X to 20X
> > slower.
> > > >>
> > > >> Check the CPU during one request. It should use nearly 100% of a
> > single
> > > >> CPU. If it a lot lower than 100%, you have another bottleneck. That
> > > might
> > > >> be insufficient heap or accessing disk during query requests (not
> > enough
> > > >> RAM). If it is near 100%, the only thing you can do is get a faster
> > CPU.
> > > >>
> > > >> One other question, how frequently is the index updated?
> > > >>
> > > >> wunder
> > > >> Walter Underwood
> > > >> [hidden email]
> > > >> http://observer.wunderwood.org/  (my blog)
> > > >>
> > > >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <
> > [hidden email]
> > > >
> > > >> wrote:
> > > >>>
> > > >>> Hi,
> > > >>> Still facing the same issue...
> > > >>> Anything else that we need to check.
> > > >>>
> > > >>>
> > > >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <
> > [hidden email]
> > > >
> > > >>> wrote:
> > > >>>
> > > >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
> > > running
> > > >>>> that combination in prod for three years with no problems.
> > > >>>>
> > > >>>> SOLR_HEAP=8g
> > > >>>> # Use G1 GC  -- wunder 2017-01-23
> > > >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
> > > >>>> GC_TUNE=" \
> > > >>>> -XX:+UseG1GC \
> > > >>>> -XX:+ParallelRefProcEnabled \
> > > >>>> -XX:G1HeapRegionSize=8m \
> > > >>>> -XX:MaxGCPauseMillis=200 \
> > > >>>> -XX:+UseLargePages \
> > > >>>> -XX:+AggressiveOpts \
> > > >>>> “
> > > >>>>
> > > >>>> wunder
> > > >>>> Walter Underwood
> > > >>>> [hidden email]
> > > >>>> http://observer.wunderwood.org/  (my blog)
> > > >>>>
> > > >>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <
> > > [hidden email]
> > > >>>
> > > >>>> wrote:
> > > >>>>>
> > > >>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space is
> 12
> > > gb.
> > > >>>> We
> > > >>>>> have completed indexing after starting the server suddenly heap
> > space
> > > >> is
> > > >>>>> getting full.
> > > >>>>> Added gc params  , still not working and jdk version is 1.8 .
> > > >>>>> Please find the below gc  params
> > > >>>>> -XX:NewRatio=2
> > > >>>>> -XX:SurvivorRatio=3
> > > >>>>> -XX:TargetSurvivorRatio=90 \
> > > >>>>> -XX:MaxTenuringThreshold=8 \
> > > >>>>> -XX:+UseConcMarkSweepGC \
> > > >>>>> -XX:+CMSScavengeBeforeRemark \
> > > >>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
> > > >>>>> -XX:PretenureSizeThreshold=512m \
> > > >>>>> -XX:CMSFullGCsBeforeCompaction=1 \
> > > >>>>> -XX:+UseCMSInitiatingOccupancyOnly \
> > > >>>>> -XX:CMSInitiatingOccupancyFraction=70 \
> > > >>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
> > > >>>>> -XX:+CMSParallelRemarkEnabled
> > > >>>>> -XX:+ParallelRefProcEnabled
> > > >>>>> -XX:+UseLargePages \
> > > >>>>> -XX:+AggressiveOpts \
> > > >>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Anything else regarding gc tuning.

On Mon, 20 Jan, 2020, 8:08 AM Rajdeep Sahoo, <[hidden email]>
wrote:

> Initially we were getting the warning message as  ulimit is low i.e. 1024
> so we changed it to 65000
> Using ulimit -u 65000.
>
> Then the error was failed to reserve shared memory error =1
>  Because of this we removed
>    -xx : +uselargepages
>
> Now in console log it is showing
> Could not find or load main class \
>
> And solr is not starting up
>
>
> On Mon, 20 Jan, 2020, 7:50 AM Mehai, Lotfi, <[hidden email]>
> wrote:
>
>> I  had a similar issue with a large number of facets. There is no way (At
>> least I know) your can get an acceptable response time from search engine
>> with high number of facets.
>> The way we solved the issue was to cache shallow Facets data structure in
>> the web services. Facts structures are refreshed periodically. We don't
>> have near real time indexation requirements. Page response time is under
>> 5s.
>>
>> Here the URLs for our worst use case:
>> https://www.govinfo.gov/app/collection/cfr
>> https://www.govinfo.gov/app/cfrparts/month
>>
>> I hope that helps.
>>
>> Lotfi Mehai
>> https://www.linkedin.com/in/lmehai/
>>
>>
>>
>>
>>
>> On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo <[hidden email]
>> >
>> wrote:
>>
>> > Initially we were getting the warning message as  ulimit is low i.e.
>> 1024
>> > so we changed it to 65000
>> > Using ulimit -u 65000.
>> >
>> > Then the error was failed to reserve shared memory error =1
>> >  Because of this we removed
>> >    -xx : +uselargepages
>> >
>> > Now in console log it is showing
>> > Could not find or load main class \
>> >
>> > And solr is not starting up
>> >
>> >
>> >
>> > On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, <[hidden email]>
>> > wrote:
>> >
>> > > What message do you get that means the heap space is full?
>> > >
>> > > Java will always use all of the heap, either as live data or
>> > > not-yet-collected garbage.
>> > >
>> > > wunder
>> > > Walter Underwood
>> > > [hidden email]
>> > > http://observer.wunderwood.org/  (my blog)
>> > >
>> > > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <
>> [hidden email]
>> > >
>> > > wrote:
>> > > >
>> > > > Hi,
>> > > > Currently there is no request or indexing is happening.
>> > > >  It's just start up
>> > > > And during that time heap is getting full.
>> > > > Index size is approx 1 g.
>> > > >
>> > > >
>> > > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <
>> [hidden email]
>> > >
>> > > > wrote:
>> > > >
>> > > >> A new garbage collector won’t fix it, but it might help a bit.
>> > > >>
>> > > >> Requesting 200 facet fields and having 50-60 of them with results
>> is a
>> > > >> huge amount of work for Solr. A typical faceting implementation
>> might
>> > > have
>> > > >> three to five facets. Your requests will be at least 10X to 20X
>> > slower.
>> > > >>
>> > > >> Check the CPU during one request. It should use nearly 100% of a
>> > single
>> > > >> CPU. If it a lot lower than 100%, you have another bottleneck. That
>> > > might
>> > > >> be insufficient heap or accessing disk during query requests (not
>> > enough
>> > > >> RAM). If it is near 100%, the only thing you can do is get a faster
>> > CPU.
>> > > >>
>> > > >> One other question, how frequently is the index updated?
>> > > >>
>> > > >> wunder
>> > > >> Walter Underwood
>> > > >> [hidden email]
>> > > >> http://observer.wunderwood.org/  (my blog)
>> > > >>
>> > > >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <
>> > [hidden email]
>> > > >
>> > > >> wrote:
>> > > >>>
>> > > >>> Hi,
>> > > >>> Still facing the same issue...
>> > > >>> Anything else that we need to check.
>> > > >>>
>> > > >>>
>> > > >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <
>> > [hidden email]
>> > > >
>> > > >>> wrote:
>> > > >>>
>> > > >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
>> > > running
>> > > >>>> that combination in prod for three years with no problems.
>> > > >>>>
>> > > >>>> SOLR_HEAP=8g
>> > > >>>> # Use G1 GC  -- wunder 2017-01-23
>> > > >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>> > > >>>> GC_TUNE=" \
>> > > >>>> -XX:+UseG1GC \
>> > > >>>> -XX:+ParallelRefProcEnabled \
>> > > >>>> -XX:G1HeapRegionSize=8m \
>> > > >>>> -XX:MaxGCPauseMillis=200 \
>> > > >>>> -XX:+UseLargePages \
>> > > >>>> -XX:+AggressiveOpts \
>> > > >>>> “
>> > > >>>>
>> > > >>>> wunder
>> > > >>>> Walter Underwood
>> > > >>>> [hidden email]
>> > > >>>> http://observer.wunderwood.org/  (my blog)
>> > > >>>>
>> > > >>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <
>> > > [hidden email]
>> > > >>>
>> > > >>>> wrote:
>> > > >>>>>
>> > > >>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space
>> is 12
>> > > gb.
>> > > >>>> We
>> > > >>>>> have completed indexing after starting the server suddenly heap
>> > space
>> > > >> is
>> > > >>>>> getting full.
>> > > >>>>> Added gc params  , still not working and jdk version is 1.8 .
>> > > >>>>> Please find the below gc  params
>> > > >>>>> -XX:NewRatio=2
>> > > >>>>> -XX:SurvivorRatio=3
>> > > >>>>> -XX:TargetSurvivorRatio=90 \
>> > > >>>>> -XX:MaxTenuringThreshold=8 \
>> > > >>>>> -XX:+UseConcMarkSweepGC \
>> > > >>>>> -XX:+CMSScavengeBeforeRemark \
>> > > >>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
>> > > >>>>> -XX:PretenureSizeThreshold=512m \
>> > > >>>>> -XX:CMSFullGCsBeforeCompaction=1 \
>> > > >>>>> -XX:+UseCMSInitiatingOccupancyOnly \
>> > > >>>>> -XX:CMSInitiatingOccupancyFraction=70 \
>> > > >>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
>> > > >>>>> -XX:+CMSParallelRemarkEnabled
>> > > >>>>> -XX:+ParallelRefProcEnabled
>> > > >>>>> -XX:+UseLargePages \
>> > > >>>>> -XX:+AggressiveOpts \
>> > > >>>>
>> > > >>>>
>> > > >>
>> > > >>
>> > >
>> > >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
The problem is happening for one index, for other two indexes it is working
fine.
  For other two indexes indexing and search both are working fine.
  But for one index after indexing completion the heap space is getting
full and solr is not responding at all.
  Index sizes are almost same,its around 1gb each.
  When loading the core by selecting it,the jvm space us becoming full.


On Mon, 20 Jan, 2020, 9:26 AM Rajdeep Sahoo, <[hidden email]>
wrote:

> Anything else regarding gc tuning.
>
> On Mon, 20 Jan, 2020, 8:08 AM Rajdeep Sahoo, <[hidden email]>
> wrote:
>
>> Initially we were getting the warning message as  ulimit is low i.e. 1024
>> so we changed it to 65000
>> Using ulimit -u 65000.
>>
>> Then the error was failed to reserve shared memory error =1
>>  Because of this we removed
>>    -xx : +uselargepages
>>
>> Now in console log it is showing
>> Could not find or load main class \
>>
>> And solr is not starting up
>>
>>
>> On Mon, 20 Jan, 2020, 7:50 AM Mehai, Lotfi, <[hidden email]>
>> wrote:
>>
>>> I  had a similar issue with a large number of facets. There is no way (At
>>> least I know) your can get an acceptable response time from search engine
>>> with high number of facets.
>>> The way we solved the issue was to cache shallow Facets data structure in
>>> the web services. Facts structures are refreshed periodically. We don't
>>> have near real time indexation requirements. Page response time is under
>>> 5s.
>>>
>>> Here the URLs for our worst use case:
>>> https://www.govinfo.gov/app/collection/cfr
>>> https://www.govinfo.gov/app/cfrparts/month
>>>
>>> I hope that helps.
>>>
>>> Lotfi Mehai
>>> https://www.linkedin.com/in/lmehai/
>>>
>>>
>>>
>>>
>>>
>>> On Sun, Jan 19, 2020 at 9:05 PM Rajdeep Sahoo <
>>> [hidden email]>
>>> wrote:
>>>
>>> > Initially we were getting the warning message as  ulimit is low i.e.
>>> 1024
>>> > so we changed it to 65000
>>> > Using ulimit -u 65000.
>>> >
>>> > Then the error was failed to reserve shared memory error =1
>>> >  Because of this we removed
>>> >    -xx : +uselargepages
>>> >
>>> > Now in console log it is showing
>>> > Could not find or load main class \
>>> >
>>> > And solr is not starting up
>>> >
>>> >
>>> >
>>> > On Mon, 20 Jan, 2020, 7:20 AM Walter Underwood, <[hidden email]
>>> >
>>> > wrote:
>>> >
>>> > > What message do you get that means the heap space is full?
>>> > >
>>> > > Java will always use all of the heap, either as live data or
>>> > > not-yet-collected garbage.
>>> > >
>>> > > wunder
>>> > > Walter Underwood
>>> > > [hidden email]
>>> > > http://observer.wunderwood.org/  (my blog)
>>> > >
>>> > > > On Jan 19, 2020, at 5:47 PM, Rajdeep Sahoo <
>>> [hidden email]
>>> > >
>>> > > wrote:
>>> > > >
>>> > > > Hi,
>>> > > > Currently there is no request or indexing is happening.
>>> > > >  It's just start up
>>> > > > And during that time heap is getting full.
>>> > > > Index size is approx 1 g.
>>> > > >
>>> > > >
>>> > > > On Mon, 20 Jan, 2020, 7:01 AM Walter Underwood, <
>>> [hidden email]
>>> > >
>>> > > > wrote:
>>> > > >
>>> > > >> A new garbage collector won’t fix it, but it might help a bit.
>>> > > >>
>>> > > >> Requesting 200 facet fields and having 50-60 of them with results
>>> is a
>>> > > >> huge amount of work for Solr. A typical faceting implementation
>>> might
>>> > > have
>>> > > >> three to five facets. Your requests will be at least 10X to 20X
>>> > slower.
>>> > > >>
>>> > > >> Check the CPU during one request. It should use nearly 100% of a
>>> > single
>>> > > >> CPU. If it a lot lower than 100%, you have another bottleneck.
>>> That
>>> > > might
>>> > > >> be insufficient heap or accessing disk during query requests (not
>>> > enough
>>> > > >> RAM). If it is near 100%, the only thing you can do is get a
>>> faster
>>> > CPU.
>>> > > >>
>>> > > >> One other question, how frequently is the index updated?
>>> > > >>
>>> > > >> wunder
>>> > > >> Walter Underwood
>>> > > >> [hidden email]
>>> > > >> http://observer.wunderwood.org/  (my blog)
>>> > > >>
>>> > > >>> On Jan 19, 2020, at 4:49 PM, Rajdeep Sahoo <
>>> > [hidden email]
>>> > > >
>>> > > >> wrote:
>>> > > >>>
>>> > > >>> Hi,
>>> > > >>> Still facing the same issue...
>>> > > >>> Anything else that we need to check.
>>> > > >>>
>>> > > >>>
>>> > > >>> On Mon, 20 Jan, 2020, 4:07 AM Walter Underwood, <
>>> > [hidden email]
>>> > > >
>>> > > >>> wrote:
>>> > > >>>
>>> > > >>>> With Java 1.8, I would use the G1 garbage collector. We’ve been
>>> > > running
>>> > > >>>> that combination in prod for three years with no problems.
>>> > > >>>>
>>> > > >>>> SOLR_HEAP=8g
>>> > > >>>> # Use G1 GC  -- wunder 2017-01-23
>>> > > >>>> # Settings from https://wiki.apache.org/solr/ShawnHeisey
>>> > > >>>> GC_TUNE=" \
>>> > > >>>> -XX:+UseG1GC \
>>> > > >>>> -XX:+ParallelRefProcEnabled \
>>> > > >>>> -XX:G1HeapRegionSize=8m \
>>> > > >>>> -XX:MaxGCPauseMillis=200 \
>>> > > >>>> -XX:+UseLargePages \
>>> > > >>>> -XX:+AggressiveOpts \
>>> > > >>>> “
>>> > > >>>>
>>> > > >>>> wunder
>>> > > >>>> Walter Underwood
>>> > > >>>> [hidden email]
>>> > > >>>> http://observer.wunderwood.org/  (my blog)
>>> > > >>>>
>>> > > >>>>> On Jan 19, 2020, at 9:25 AM, Rajdeep Sahoo <
>>> > > [hidden email]
>>> > > >>>
>>> > > >>>> wrote:
>>> > > >>>>>
>>> > > >>>>> We are using solr 7.7 . Ram size is 24 gb and allocated space
>>> is 12
>>> > > gb.
>>> > > >>>> We
>>> > > >>>>> have completed indexing after starting the server suddenly heap
>>> > space
>>> > > >> is
>>> > > >>>>> getting full.
>>> > > >>>>> Added gc params  , still not working and jdk version is 1.8 .
>>> > > >>>>> Please find the below gc  params
>>> > > >>>>> -XX:NewRatio=2
>>> > > >>>>> -XX:SurvivorRatio=3
>>> > > >>>>> -XX:TargetSurvivorRatio=90 \
>>> > > >>>>> -XX:MaxTenuringThreshold=8 \
>>> > > >>>>> -XX:+UseConcMarkSweepGC \
>>> > > >>>>> -XX:+CMSScavengeBeforeRemark \
>>> > > >>>>> -XX:ConcGCThreads=4 -XX:ParallelGCThreads=4 \
>>> > > >>>>> -XX:PretenureSizeThreshold=512m \
>>> > > >>>>> -XX:CMSFullGCsBeforeCompaction=1 \
>>> > > >>>>> -XX:+UseCMSInitiatingOccupancyOnly \
>>> > > >>>>> -XX:CMSInitiatingOccupancyFraction=70 \
>>> > > >>>>> -XX:CMSMaxAbortablePrecleanTime=6000 \
>>> > > >>>>> -XX:+CMSParallelRemarkEnabled
>>> > > >>>>> -XX:+ParallelRefProcEnabled
>>> > > >>>>> -XX:+UseLargePages \
>>> > > >>>>> -XX:+AggressiveOpts \
>>> > > >>>>
>>> > > >>>>
>>> > > >>
>>> > > >>
>>> > >
>>> > >
>>> >
>>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Toke Eskildsen-2
In reply to this post by Mehai, Lotfi
On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> I  had a similar issue with a large number of facets. There is no way
> (At least I know) your can get an acceptable response time from
> search engine with high number of facets.

Just for the record then it is doable under specific circumstances
(static single-shard index, only String fields, Solr 4 with patch,
fixed list of facet fields):
https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/

More usable for the current case would be to play with facet.threads
and throw hardware with many CPU-cores after the problem.

- Toke Eskildsen, Royal Danish Library


Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Michael Gibney
Rajdeep, you say that "suddenly" heap space is getting full ... does
this mean that some variant of this configuration was working for you
at some point, or just that the failure happens quickly?

If heap space and faceting are indeed the bottleneck, you might make
sure that you have docValues enabled for your facet field fieldTypes,
and perhaps set uninvertible=false.

I'm not seeing where large numbers of facets initially came from in
this thread? But on that topic this is perhaps relevant, regarding the
potential utility of a facet cache:
https://issues.apache.org/jira/browse/SOLR-13807

Michael

On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:

>
> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> > I  had a similar issue with a large number of facets. There is no way
> > (At least I know) your can get an acceptable response time from
> > search engine with high number of facets.
>
> Just for the record then it is doable under specific circumstances
> (static single-shard index, only String fields, Solr 4 with patch,
> fixed list of facet fields):
> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
>
> More usable for the current case would be to play with facet.threads
> and throw hardware with many CPU-cores after the problem.
>
> - Toke Eskildsen, Royal Danish Library
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
This is happening when the no of indexed document count is increasing.
   With 1 million docs it's working fine but when it's crossing 4.5 million
it's heap space is getting full.


On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <[hidden email]>
wrote:

> Rajdeep, you say that "suddenly" heap space is getting full ... does
> this mean that some variant of this configuration was working for you
> at some point, or just that the failure happens quickly?
>
> If heap space and faceting are indeed the bottleneck, you might make
> sure that you have docValues enabled for your facet field fieldTypes,
> and perhaps set uninvertible=false.
>
> I'm not seeing where large numbers of facets initially came from in
> this thread? But on that topic this is perhaps relevant, regarding the
> potential utility of a facet cache:
> https://issues.apache.org/jira/browse/SOLR-13807
>
> Michael
>
> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:
> >
> > On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> > > I  had a similar issue with a large number of facets. There is no way
> > > (At least I know) your can get an acceptable response time from
> > > search engine with high number of facets.
> >
> > Just for the record then it is doable under specific circumstances
> > (static single-shard index, only String fields, Solr 4 with patch,
> > fixed list of facet fields):
> > https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
> >
> > More usable for the current case would be to play with facet.threads
> > and throw hardware with many CPU-cores after the problem.
> >
> > - Toke Eskildsen, Royal Danish Library
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
Please reply anyone

On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <[hidden email]>
wrote:

> This is happening when the no of indexed document count is increasing.
>    With 1 million docs it's working fine but when it's crossing 4.5
> million it's heap space is getting full.
>
>
> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <[hidden email]>
> wrote:
>
>> Rajdeep, you say that "suddenly" heap space is getting full ... does
>> this mean that some variant of this configuration was working for you
>> at some point, or just that the failure happens quickly?
>>
>> If heap space and faceting are indeed the bottleneck, you might make
>> sure that you have docValues enabled for your facet field fieldTypes,
>> and perhaps set uninvertible=false.
>>
>> I'm not seeing where large numbers of facets initially came from in
>> this thread? But on that topic this is perhaps relevant, regarding the
>> potential utility of a facet cache:
>> https://issues.apache.org/jira/browse/SOLR-13807
>>
>> Michael
>>
>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:
>> >
>> > On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
>> > > I  had a similar issue with a large number of facets. There is no way
>> > > (At least I know) your can get an acceptable response time from
>> > > search engine with high number of facets.
>> >
>> > Just for the record then it is doable under specific circumstances
>> > (static single-shard index, only String fields, Solr 4 with patch,
>> > fixed list of facet fields):
>> > https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
>> >
>> > More usable for the current case would be to play with facet.threads
>> > and throw hardware with many CPU-cores after the problem.
>> >
>> > - Toke Eskildsen, Royal Danish Library
>> >
>> >
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Walter Underwood
What message do you get about the heap space.

It is completely normal for Java to use all of heap before running a major GC. That
is how the JVM works.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Feb 1, 2020, at 6:35 AM, Rajdeep Sahoo <[hidden email]> wrote:
>
> Please reply anyone
>
> On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <[hidden email]>
> wrote:
>
>> This is happening when the no of indexed document count is increasing.
>>   With 1 million docs it's working fine but when it's crossing 4.5
>> million it's heap space is getting full.
>>
>>
>> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <[hidden email]>
>> wrote:
>>
>>> Rajdeep, you say that "suddenly" heap space is getting full ... does
>>> this mean that some variant of this configuration was working for you
>>> at some point, or just that the failure happens quickly?
>>>
>>> If heap space and faceting are indeed the bottleneck, you might make
>>> sure that you have docValues enabled for your facet field fieldTypes,
>>> and perhaps set uninvertible=false.
>>>
>>> I'm not seeing where large numbers of facets initially came from in
>>> this thread? But on that topic this is perhaps relevant, regarding the
>>> potential utility of a facet cache:
>>> https://issues.apache.org/jira/browse/SOLR-13807
>>>
>>> Michael
>>>
>>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:
>>>>
>>>> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
>>>>> I  had a similar issue with a large number of facets. There is no way
>>>>> (At least I know) your can get an acceptable response time from
>>>>> search engine with high number of facets.
>>>>
>>>> Just for the record then it is doable under specific circumstances
>>>> (static single-shard index, only String fields, Solr 4 with patch,
>>>> fixed list of facet fields):
>>>> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
>>>>
>>>> More usable for the current case would be to play with facet.threads
>>>> and throw hardware with many CPU-cores after the problem.
>>>>
>>>> - Toke Eskildsen, Royal Danish Library
>>>>
>>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
What can we do in this scenario as the solr master node is going down and
the indexing is failing.
 Please provide some workaround for this issue.

On Sat, 1 Feb, 2020, 11:51 PM Walter Underwood, <[hidden email]>
wrote:

> What message do you get about the heap space.
>
> It is completely normal for Java to use all of heap before running a major
> GC. That
> is how the JVM works.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Feb 1, 2020, at 6:35 AM, Rajdeep Sahoo <[hidden email]>
> wrote:
> >
> > Please reply anyone
> >
> > On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <
> [hidden email]>
> > wrote:
> >
> >> This is happening when the no of indexed document count is increasing.
> >>   With 1 million docs it's working fine but when it's crossing 4.5
> >> million it's heap space is getting full.
> >>
> >>
> >> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <
> [hidden email]>
> >> wrote:
> >>
> >>> Rajdeep, you say that "suddenly" heap space is getting full ... does
> >>> this mean that some variant of this configuration was working for you
> >>> at some point, or just that the failure happens quickly?
> >>>
> >>> If heap space and faceting are indeed the bottleneck, you might make
> >>> sure that you have docValues enabled for your facet field fieldTypes,
> >>> and perhaps set uninvertible=false.
> >>>
> >>> I'm not seeing where large numbers of facets initially came from in
> >>> this thread? But on that topic this is perhaps relevant, regarding the
> >>> potential utility of a facet cache:
> >>> https://issues.apache.org/jira/browse/SOLR-13807
> >>>
> >>> Michael
> >>>
> >>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:
> >>>>
> >>>> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> >>>>> I  had a similar issue with a large number of facets. There is no way
> >>>>> (At least I know) your can get an acceptable response time from
> >>>>> search engine with high number of facets.
> >>>>
> >>>> Just for the record then it is doable under specific circumstances
> >>>> (static single-shard index, only String fields, Solr 4 with patch,
> >>>> fixed list of facet fields):
> >>>> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
> >>>>
> >>>> More usable for the current case would be to play with facet.threads
> >>>> and throw hardware with many CPU-cores after the problem.
> >>>>
> >>>> - Toke Eskildsen, Royal Danish Library
> >>>>
> >>>>
> >>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Erick Erickson
Allocate more heap and possibly add more RAM.

What are you expectations? You can't continue to
add documents to your Solr instance without regard to
how much heap you’ve allocated. You’ve put over 4x
the number of docs on the node. There’s no magic here.
You can’t continue to add docs to a Solr instance without
increasing the heap at some point.

And as far as I know, you’ve never told us how much heap yo
 _are_ allocating. The default for Java processes is 512M, which
is quite small. so perhaps it’s a simple matter of starting Solr
with the -XmX parameter set to something larger.

Best,
Erick

> On Feb 2, 2020, at 10:19 AM, Rajdeep Sahoo <[hidden email]> wrote:
>
> What can we do in this scenario as the solr master node is going down and
> the indexing is failing.
> Please provide some workaround for this issue.
>
> On Sat, 1 Feb, 2020, 11:51 PM Walter Underwood, <[hidden email]>
> wrote:
>
>> What message do you get about the heap space.
>>
>> It is completely normal for Java to use all of heap before running a major
>> GC. That
>> is how the JVM works.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Feb 1, 2020, at 6:35 AM, Rajdeep Sahoo <[hidden email]>
>> wrote:
>>>
>>> Please reply anyone
>>>
>>> On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <
>> [hidden email]>
>>> wrote:
>>>
>>>> This is happening when the no of indexed document count is increasing.
>>>>  With 1 million docs it's working fine but when it's crossing 4.5
>>>> million it's heap space is getting full.
>>>>
>>>>
>>>> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <
>> [hidden email]>
>>>> wrote:
>>>>
>>>>> Rajdeep, you say that "suddenly" heap space is getting full ... does
>>>>> this mean that some variant of this configuration was working for you
>>>>> at some point, or just that the failure happens quickly?
>>>>>
>>>>> If heap space and faceting are indeed the bottleneck, you might make
>>>>> sure that you have docValues enabled for your facet field fieldTypes,
>>>>> and perhaps set uninvertible=false.
>>>>>
>>>>> I'm not seeing where large numbers of facets initially came from in
>>>>> this thread? But on that topic this is perhaps relevant, regarding the
>>>>> potential utility of a facet cache:
>>>>> https://issues.apache.org/jira/browse/SOLR-13807
>>>>>
>>>>> Michael
>>>>>
>>>>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:
>>>>>>
>>>>>> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
>>>>>>> I  had a similar issue with a large number of facets. There is no way
>>>>>>> (At least I know) your can get an acceptable response time from
>>>>>>> search engine with high number of facets.
>>>>>>
>>>>>> Just for the record then it is doable under specific circumstances
>>>>>> (static single-shard index, only String fields, Solr 4 with patch,
>>>>>> fixed list of facet fields):
>>>>>> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
>>>>>>
>>>>>> More usable for the current case would be to play with facet.threads
>>>>>> and throw hardware with many CPU-cores after the problem.
>>>>>>
>>>>>> - Toke Eskildsen, Royal Danish Library
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.7 heap space is getting full

Rajdeep Sahoo
We have allocated 16 gb of heap space  out of 24 g.
   There are 3 solr cores here, for one core when the no of documents are
getting increased i.e. around 4.5 lakhs,then this scenario is happening.


On Sun, 2 Feb, 2020, 9:02 PM Erick Erickson, <[hidden email]>
wrote:

> Allocate more heap and possibly add more RAM.
>
> What are you expectations? You can't continue to
> add documents to your Solr instance without regard to
> how much heap you’ve allocated. You’ve put over 4x
> the number of docs on the node. There’s no magic here.
> You can’t continue to add docs to a Solr instance without
> increasing the heap at some point.
>
> And as far as I know, you’ve never told us how much heap yo
>  _are_ allocating. The default for Java processes is 512M, which
> is quite small. so perhaps it’s a simple matter of starting Solr
> with the -XmX parameter set to something larger.
>
> Best,
> Erick
>
> > On Feb 2, 2020, at 10:19 AM, Rajdeep Sahoo <[hidden email]>
> wrote:
> >
> > What can we do in this scenario as the solr master node is going down and
> > the indexing is failing.
> > Please provide some workaround for this issue.
> >
> > On Sat, 1 Feb, 2020, 11:51 PM Walter Underwood, <[hidden email]>
> > wrote:
> >
> >> What message do you get about the heap space.
> >>
> >> It is completely normal for Java to use all of heap before running a
> major
> >> GC. That
> >> is how the JVM works.
> >>
> >> wunder
> >> Walter Underwood
> >> [hidden email]
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>> On Feb 1, 2020, at 6:35 AM, Rajdeep Sahoo <[hidden email]>
> >> wrote:
> >>>
> >>> Please reply anyone
> >>>
> >>> On Fri, 31 Jan, 2020, 11:37 PM Rajdeep Sahoo, <
> >> [hidden email]>
> >>> wrote:
> >>>
> >>>> This is happening when the no of indexed document count is increasing.
> >>>>  With 1 million docs it's working fine but when it's crossing 4.5
> >>>> million it's heap space is getting full.
> >>>>
> >>>>
> >>>> On Wed, 22 Jan, 2020, 7:05 PM Michael Gibney, <
> >> [hidden email]>
> >>>> wrote:
> >>>>
> >>>>> Rajdeep, you say that "suddenly" heap space is getting full ... does
> >>>>> this mean that some variant of this configuration was working for you
> >>>>> at some point, or just that the failure happens quickly?
> >>>>>
> >>>>> If heap space and faceting are indeed the bottleneck, you might make
> >>>>> sure that you have docValues enabled for your facet field fieldTypes,
> >>>>> and perhaps set uninvertible=false.
> >>>>>
> >>>>> I'm not seeing where large numbers of facets initially came from in
> >>>>> this thread? But on that topic this is perhaps relevant, regarding
> the
> >>>>> potential utility of a facet cache:
> >>>>> https://issues.apache.org/jira/browse/SOLR-13807
> >>>>>
> >>>>> Michael
> >>>>>
> >>>>> On Wed, Jan 22, 2020 at 7:16 AM Toke Eskildsen <[hidden email]> wrote:
> >>>>>>
> >>>>>> On Sun, 2020-01-19 at 21:19 -0500, Mehai, Lotfi wrote:
> >>>>>>> I  had a similar issue with a large number of facets. There is no
> way
> >>>>>>> (At least I know) your can get an acceptable response time from
> >>>>>>> search engine with high number of facets.
> >>>>>>
> >>>>>> Just for the record then it is doable under specific circumstances
> >>>>>> (static single-shard index, only String fields, Solr 4 with patch,
> >>>>>> fixed list of facet fields):
> >>>>>> https://sbdevel.wordpress.com/2013/03/20/over-9000-facet-fields/
> >>>>>>
> >>>>>> More usable for the current case would be to play with facet.threads
> >>>>>> and throw hardware with many CPU-cores after the problem.
> >>>>>>
> >>>>>> - Toke Eskildsen, Royal Danish Library
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>
> >>
>
>
12