Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

mganeshs
Solr experts,

We found following  link
<https://www.google.com/url?q=http://lucene.472066.n3.nabble.com/CommitScheduler-Thread-blocked-due-to-excessive-number-of-Merging-Threads-td4353964.html&sa=D&source=hangouts&ust=1523527480128000&usg=AFQjCNENaNhq49nMSj8WtS_Qx-N4K23v0Q>  
where its mentioned like in 6.2.1 it's faster where as in 6.6 its slower.

We are also facing same issue...with 6.2.1 in our performance environment
and we
found that CPU usage is around 60 to 70% where as in 6.5.1 it was always
more than 95%

Settings are same and data size and indexing speed remains same. Pls check
the  JVM snapshot
<https://lh3.googleusercontent.com/-MHYsop5Vovo/Ws3V8KrT2VI/AAAAAAAAALI/GXmcWA4OtPQwwpF09scU5riJ1VHFOUtHQCL0BGAYYCw/h1080/2018-04-11.png>  
when we index using 6.2.1


Following is the  snapshot
<https://lh3.googleusercontent.com/-FoOmxYACbKc/Ws3gUQ323rI/AAAAAAAACeA/xdZedHlr1pcRbBrxhNJFPSvZqaH6rIoEACL0BGAYYCw/h363/2018-04-11.png>
taken with 6.5.1

Is there any reason why such a huge difference with CPU usage patterns
between 6.2.1 and 6.5.1 ?

Can we do something in 6.5.1 to make it as 6.2.1? Because we don't want to
downgrade to 6.2.1 from 6.5.1.

Let us know your thoughts on this.

Thanks and Regards,




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Bernd Fehling
As a first guess I would say that you have much higher GC activity
which causes much higher CPU usage.
Why do you have much higher GC activity?
Any GC settings changed?
Have you tried increasing heap size?

Regards
Bernd


Am 16.04.2018 um 06:22 schrieb mganeshs:

> Solr experts,
>
> We found following  link
> <https://www.google.com/url?q=http://lucene.472066.n3.nabble.com/CommitScheduler-Thread-blocked-due-to-excessive-number-of-Merging-Threads-td4353964.html&sa=D&source=hangouts&ust=1523527480128000&usg=AFQjCNENaNhq49nMSj8WtS_Qx-N4K23v0Q>  
> where its mentioned like in 6.2.1 it's faster where as in 6.6 its slower.
>
> We are also facing same issue...with 6.2.1 in our performance environment
> and we
> found that CPU usage is around 60 to 70% where as in 6.5.1 it was always
> more than 95%
>
> Settings are same and data size and indexing speed remains same. Pls check
> the  JVM snapshot
> <https://lh3.googleusercontent.com/-MHYsop5Vovo/Ws3V8KrT2VI/AAAAAAAAALI/GXmcWA4OtPQwwpF09scU5riJ1VHFOUtHQCL0BGAYYCw/h1080/2018-04-11.png>  
> when we index using 6.2.1
>
>
> Following is the  snapshot
> <https://lh3.googleusercontent.com/-FoOmxYACbKc/Ws3gUQ323rI/AAAAAAAACeA/xdZedHlr1pcRbBrxhNJFPSvZqaH6rIoEACL0BGAYYCw/h363/2018-04-11.png>
> taken with 6.5.1
>
> Is there any reason why such a huge difference with CPU usage patterns
> between 6.2.1 and 6.5.1 ?
>
> Can we do something in 6.5.1 to make it as 6.2.1? Because we don't want to
> downgrade to 6.2.1 from 6.5.1.
>
> Let us know your thoughts on this.
>
> Thanks and Regards,
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

mganeshs
Hi Bernd,

We didn't change any default settings.

Both 6.2.1 and 6.5.1 is running with same settings, same volume of data,
same code, which means indexing rate is also same.

In Case of 6.2.1 CPU is around 60 to 70%. But in 6.5.1 it's always around
95%. The CPU % in 6.5.1 is alarming for us and we keep getting alerts as
it's always more than 95%.

Basically, my question is why is that in 6.2.1 CPU is low and for 6.5.1 it's
very high ? I though only I am facing this issue, but one more in the forum
also raised this issue, but nothing concluded so far.

In another thread Shawn also suggested changes wrt merge policy numbers. But
CPU % didn't come down. But in 6.2.1 with default settings itself, it works
fine and CPU is also normal. So created new thread to discuss wrt CPU
utilization between old version (6.2.1 ) and new version (6.5.1+)

Regards,



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Bernd Fehling
In reply to this post by Bernd Fehling
It would help if you can trace it down to a version change.
Do you have a test system and start with 6.3.0 as next version above 6.2.1
to see which version change is making you trouble?
You can then try 6.4.0 and 6.5.0 next. And after that go into subversions.

Regards, Bernd


Am 16.04.2018 um 09:39 schrieb mganeshs:

> Hi Bernd,
>
> We didn't change any default settings.
>
> Both 6.2.1 and 6.5.1 is running with same settings, same volume of data,
> same code, which means indexing rate is also same.
>
> In Case of 6.2.1 CPU is around 60 to 70%. But in 6.5.1 it's always around
> 95%. The CPU % in 6.5.1 is alarming for us and we keep getting alerts as
> it's always more than 95%.
>
> Basically, my question is why is that in 6.2.1 CPU is low and for 6.5.1 it's
> very high ? I though only I am facing this issue, but one more in the forum
> also raised this issue, but nothing concluded so far.
>
> In another thread Shawn also suggested changes wrt merge policy numbers. But
> CPU % didn't come down. But in 6.2.1 with default settings itself, it works
> fine and CPU is also normal. So created new thread to discuss wrt CPU
> utilization between old version (6.2.1 ) and new version (6.5.1+)
>
> Regards,
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Deepak Goel
Do you see a performance improvement in your 'Query Times'  (6.2.1
vis-a-vis 6.5.1)?



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Mon, Apr 16, 2018 at 3:15 PM, Bernd Fehling <
[hidden email]> wrote:

> It would help if you can trace it down to a version change.
> Do you have a test system and start with 6.3.0 as next version above 6.2.1
> to see which version change is making you trouble?
> You can then try 6.4.0 and 6.5.0 next. And after that go into subversions.
>
> Regards, Bernd
>
>
> Am 16.04.2018 um 09:39 schrieb mganeshs:
> > Hi Bernd,
> >
> > We didn't change any default settings.
> >
> > Both 6.2.1 and 6.5.1 is running with same settings, same volume of data,
> > same code, which means indexing rate is also same.
> >
> > In Case of 6.2.1 CPU is around 60 to 70%. But in 6.5.1 it's always around
> > 95%. The CPU % in 6.5.1 is alarming for us and we keep getting alerts as
> > it's always more than 95%.
> >
> > Basically, my question is why is that in 6.2.1 CPU is low and for 6.5.1
> it's
> > very high ? I though only I am facing this issue, but one more in the
> forum
> > also raised this issue, but nothing concluded so far.
> >
> > In another thread Shawn also suggested changes wrt merge policy numbers.
> But
> > CPU % didn't come down. But in 6.2.1 with default settings itself, it
> works
> > fine and CPU is also normal. So created new thread to discuss wrt CPU
> > utilization between old version (6.2.1 ) and new version (6.5.1+)
> >
> > Regards,
> >
> >
> >
> > --
> > Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Walter Underwood
The metrics code added in 6.4 cause a big performance problem. I believe that was fixed by 6.5.1, but there might be some lingering issues.

See: https://issues.apache.org/jira/browse/SOLR-10130

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Apr 16, 2018, at 5:55 AM, Deepak Goel <[hidden email]> wrote:
>
> Do you see a performance improvement in your 'Query Times'  (6.2.1
> vis-a-vis 6.5.1)?
>
>
>
> Deepak
> "Please stop cruelty to Animals, help by becoming a Vegan"
> +91 73500 12833
> [hidden email]
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> On Mon, Apr 16, 2018 at 3:15 PM, Bernd Fehling <
> [hidden email]> wrote:
>
>> It would help if you can trace it down to a version change.
>> Do you have a test system and start with 6.3.0 as next version above 6.2.1
>> to see which version change is making you trouble?
>> You can then try 6.4.0 and 6.5.0 next. And after that go into subversions.
>>
>> Regards, Bernd
>>
>>
>> Am 16.04.2018 um 09:39 schrieb mganeshs:
>>> Hi Bernd,
>>>
>>> We didn't change any default settings.
>>>
>>> Both 6.2.1 and 6.5.1 is running with same settings, same volume of data,
>>> same code, which means indexing rate is also same.
>>>
>>> In Case of 6.2.1 CPU is around 60 to 70%. But in 6.5.1 it's always around
>>> 95%. The CPU % in 6.5.1 is alarming for us and we keep getting alerts as
>>> it's always more than 95%.
>>>
>>> Basically, my question is why is that in 6.2.1 CPU is low and for 6.5.1
>> it's
>>> very high ? I though only I am facing this issue, but one more in the
>> forum
>>> also raised this issue, but nothing concluded so far.
>>>
>>> In another thread Shawn also suggested changes wrt merge policy numbers.
>> But
>>> CPU % didn't come down. But in 6.2.1 with default settings itself, it
>> works
>>> fine and CPU is also normal. So created new thread to discuss wrt CPU
>>> utilization between old version (6.2.1 ) and new version (6.5.1+)
>>>
>>> Regards,
>>>
>>>
>>>
>>> --
>>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

mganeshs
In reply to this post by Deepak Goel
Regarding query times, we couldn't see big improvements. Both are more or
less same.

Our main worry is that, why CPU usage is so high in 6.5.1 and above ? What's
going wrong ?

Is any one else facing this sort of issue ? If yes, how to bring down the
CPU usage? Is there any settings which we need to set ( not default one ) in
6.5.1 ?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Deepak Goel
In reply to this post by mganeshs
Please post the exact results. Many a times the high cpu utilisation may be
a boon as it improves query response times

On Tue, 17 Apr 2018, 13:55 mganeshs, <[hidden email]> wrote:

> Regarding query times, we couldn't see big improvements. Both are more or
> less same.
>
> Our main worry is that, why CPU usage is so high in 6.5.1 and above ?
> What's
> going wrong ?
>
> Is any one else facing this sort of issue ? If yes, how to bring down the
> CPU usage? Is there any settings which we need to set ( not default one )
> in
> 6.5.1 ?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

mganeshs
Hello Deepak,

We are not querying when indexing is going on. Whatever CPU graph I shared
for 6.2.1 and 6.5.1 was only while we do batch indexing. During that time we
don't query and no queries are getting executed.

We index in a batch with a rate of around 100 documents / sec. And it's not
so high too. But same piece of code and same config, with 6.2.1 CPU is
normal and in 6.5.1 it always stays above 90% or 95%.

@Solr Experts,

From one of the thread by " Yasoob
<http://lucene.472066.n3.nabble.com/CommitScheduler-Thread-blocked-due-to-excessive-number-of-Merging-Threads-tp4353964p4354334.html>
" it's mentioned as

/I compared the source code for the two versions and found that different
merge functions were being used to merge the postings. In 5.4, the default
merge method of FieldsConsumer class was being used. While in 6.6, the
PerFieldPostingsFormat's merge method is being used. I checked and it looks
like this change went in Solr 6.3. So I replaced the 6.6 instance with 6.2.1
and re-indexed all the data, and it is working very well, even with the
settings I had initially used. /

Is anyone else facing this issue or any fixes got released in future build
for this ?

Keep us posted....


Deepak Goel wrote
> Please post the exact results. Many a times the high cpu utilisation may
> be
> a boon as it improves query response times
>
> On Tue, 17 Apr 2018, 13:55 mganeshs, &lt;

> mganeshs@

> &gt; wrote:
>
>> Regarding query times, we couldn't see big improvements. Both are more or
>> less same.
>>
>> Our main worry is that, why CPU usage is so high in 6.5.1 and above ?
>> What's
>> going wrong ?
>>
>> Is any one else facing this sort of issue ? If yes, how to bring down the
>> CPU usage? Is there any settings which we need to set ( not default one )
>> in
>> 6.5.1 ?
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>


Deepak Goel wrote
> Please post the exact results. Many a times the high cpu utilisation may
> be
> a boon as it improves query response times
>
> On Tue, 17 Apr 2018, 13:55 mganeshs, &lt;

> mganeshs@

> &gt; wrote:
>
>> Regarding query times, we couldn't see big improvements. Both are more or
>> less same.
>>
>> Our main worry is that, why CPU usage is so high in 6.5.1 and above ?
>> What's
>> going wrong ?
>>
>> Is any one else facing this sort of issue ? If yes, how to bring down the
>> CPU usage? Is there any settings which we need to set ( not default one )
>> in
>> 6.5.1 ?
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Performance & CPU Usage of 6.2.1 vs 6.5.1 & above

Deepak Goel
Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Thu, Apr 19, 2018 at 9:23 AM, mganeshs <[hidden email]> wrote:

> Hello Deepak,
>
> We are not querying when indexing is going on. Whatever CPU graph I shared
> for 6.2.1 and 6.5.1 was only while we do batch indexing. During that time
> we
> don't query and no queries are getting executed.
>
> We index in a batch with a rate of around 100 documents / sec.


Is the batch rate same for both the cases (6.2.1 & 6.5.1)?


> And it's not
> so high too. But same piece of code and same config, with 6.2.1 CPU is
> normal and in 6.5.1 it always stays above 90% or 95%.
>
> @Solr Experts,
>
> From one of the thread by " Yasoob
> <http://lucene.472066.n3.nabble.com/CommitScheduler-Thread-blocked-due-to-
> excessive-number-of-Merging-Threads-tp4353964p4354334.html>
> " it's mentioned as
>
> /I compared the source code for the two versions and found that different
> merge functions were being used to merge the postings. In 5.4, the default
> merge method of FieldsConsumer class was being used. While in 6.6, the
> PerFieldPostingsFormat's merge method is being used. I checked and it
> looks
> like this change went in Solr 6.3. So I replaced the 6.6 instance with
> 6.2.1
> and re-indexed all the data, and it is working very well, even with the
> settings I had initially used. /
>
> Is anyone else facing this issue or any fixes got released in future build
> for this ?
>
> Keep us posted....
>
>
> Deepak Goel wrote
> > Please post the exact results. Many a times the high cpu utilisation may
> > be
> > a boon as it improves query response times
> >
> > On Tue, 17 Apr 2018, 13:55 mganeshs, &lt;
>
> > mganeshs@
>
> > &gt; wrote:
> >
> >> Regarding query times, we couldn't see big improvements. Both are more
> or
> >> less same.
> >>
> >> Our main worry is that, why CPU usage is so high in 6.5.1 and above ?
> >> What's
> >> going wrong ?
> >>
> >> Is any one else facing this sort of issue ? If yes, how to bring down
> the
> >> CPU usage? Is there any settings which we need to set ( not default one
> )
> >> in
> >> 6.5.1 ?
> >>
> >>
> >>
> >> --
> >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >>
>
>
> Deepak Goel wrote
> > Please post the exact results. Many a times the high cpu utilisation may
> > be
> > a boon as it improves query response times
> >
> > On Tue, 17 Apr 2018, 13:55 mganeshs, &lt;
>
> > mganeshs@
>
> > &gt; wrote:
> >
> >> Regarding query times, we couldn't see big improvements. Both are more
> or
> >> less same.
> >>
> >> Our main worry is that, why CPU usage is so high in 6.5.1 and above ?
> >> What's
> >> going wrong ?
> >>
> >> Is any one else facing this sort of issue ? If yes, how to bring down
> the
> >> CPU usage? Is there any settings which we need to set ( not default one
> )
> >> in
> >> 6.5.1 ?
> >>
> >>
> >>
> >> --
> >> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
> >>
>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>