Solr4 To Solr6 CPU load issues

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Solr4 To Solr6 CPU load issues

~$alpha`
I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but
overall the CPU load on solr6.6 is double the load on solr4.0 and hence I am
not able to roll solr6.6 to 100% of my traffic.

*Some Key Stats In Performance of Sol6 Vs Solr4*
Document cache usage increased from .98 from .14
Query Result cache usage increased from .10 from .24
Filter cache same as .94
Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field
multivalued concept was changed from solr4 to solr6)

*Please Help Note: I have given document cache 3 times memory for
doc.cache.*





--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
Can you please give the configuration of your server?

On 11 Feb 2018 19:17, "~$alpha`" <[hidden email]> wrote:

> I have upgraded Solr4.0 Beta to Solr6.6. The Cache results look Awesome but
> overall the CPU load on solr6.6 is double the load on solr4.0 and hence I
> am
> not able to roll solr6.6 to 100% of my traffic.
>
> *Some Key Stats In Performance of Sol6 Vs Solr4*
> Document cache usage increased from .98 from .14
> Query Result cache usage increased from .10 from .24
> Filter cache same as .94
> Field Value cache was 0.99 in solr4 but n/a in solr6 (i guess because field
> multivalued concept was changed from solr4 to solr6)
>
> *Please Help Note: I have given document cache 3 times memory for
> doc.cache.*
>
>
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
This post was updated on .
Config : 64GB RAM 32 CORE CPU
but i have given 20Gb to solr JVM..



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
Two things trouble me:

1. It is a shared resource, so results are unreliable

2. Since cache results have increased, memory access will increase and it
will result in an increase in cpu usage. However response times will also
improve

To support more load you will have to increase server capacity or add
servers.




On 11 Feb 2018 23:28, "~$alpha`" <[hidden email]> wrote:

Config : 64GB RAM 32 CORE CPU
but i have given 20Gb to solr JVM.. Also its a shared resource



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
Other resource is not using the cpu.
Its true that response is better to 300ms from 350ms but cpu usage almost
doubled?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
Yup. Improvement of response time would hurt the cpu usage. The other thing
is more memory usage (cache) which gets included into the cpu usage.



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Mon, Feb 12, 2018 at 12:24 AM, ~$alpha` <[hidden email]> wrote:

> Other resource is not using the cpu.
> Its true that response is better to 300ms from 350ms but cpu usage almost
> doubled?
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
Also can you please post the throughputs for both of your tests

On 12 Feb 2018 00:35, "Deepak Goel" <[hidden email]> wrote:

> Yup. Improvement of response time would hurt the cpu usage. The other
> thing is more memory usage (cache) which gets included into the cpu usage.
>
>
>
> Deepak
> "Please stop cruelty to Animals, help by becoming a Vegan"
> +91 73500 12833
> [hidden email]
>
> Facebook: https://www.facebook.com/deicool
> LinkedIn: www.linkedin.com/in/deicool
>
> "Plant a Tree, Go Green"
>
> On Mon, Feb 12, 2018 at 12:24 AM, ~$alpha` <[hidden email]> wrote:
>
>> Other resource is not using the cpu.
>> Its true that response is better to 300ms from 350ms but cpu usage almost
>> doubled?
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
When both solr4 and solr6 have concurrent hits:
1. 30 to 40 :
Avg response time 470ms vs 380ms
Load                     6 vs 10

1. 80 to 90 :
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
Load                     11 vs 25




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
I would suggest to keep the load same both for solr4 and solr6. And then
test. Also please post exact concurrent hits

On 12 Feb 2018 12:48, "~$alpha`" <[hidden email]> wrote:

When both solr4 and solr6 have concurrent hits:
1. 30 to 40 :
Avg response time 470ms vs 380ms
Load                     6 vs 10

1. 80 to 90 :
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
Load                     11 vs 25




--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
Hits 41 :
Avg response time                    470ms vs 380ms
CPU Load  reaches                    6 vs 10

Hits 82:
Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
CPU Load  reaches                    11 vs 25



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
Please test for higher number of hits till cpu load reaches 100%

On 12 Feb 2018 19:44, "~$alpha`" <[hidden email]> wrote:

> Hits 41 :
> Avg response time                    470ms vs 380ms
> CPU Load  reaches                    6 vs 10
>
> Hits 82:
> Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
> CPU Load  reaches                    11 vs 25
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
I cant test on more as performance is already degraded.
Its a 32core system and load 25 means 2500% cpu



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Walter Underwood
In reply to this post by Deepak Goel
100% CPU will cause congestion and very slow response.

In production, we do not drive Solr over 75% CPU.

He is reporting load average, which is a bit harder to interpret. When the load average reaches the number of CPUs, that is probably the beginning of congestion. But I’m less sure about that than about the 75% CPU level.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Feb 12, 2018, at 6:17 AM, Deepak Goel <[hidden email]> wrote:
>
> Please test for higher number of hits till cpu load reaches 100%
>
> On 12 Feb 2018 19:44, "~$alpha`" <[hidden email]> wrote:
>
>> Hits 41 :
>> Avg response time                    470ms vs 380ms
>> CPU Load  reaches                    6 vs 10
>>
>> Hits 82:
>> Avg response time 500ms vs 620ms (solr6 performing bad on peak hours)
>> CPU Load  reaches                    11 vs 25
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>

Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
In reply to this post by ~$alpha`
This would then mean that solr6 is reaching some kind of saturation (number
of threads, etc) at about loads of 60 Hits which then drives the
performance of it to be very bad !



Deepak
"Please stop cruelty to Animals, help by becoming a Vegan"
+91 73500 12833
[hidden email]

Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool

"Plant a Tree, Go Green"

On Mon, Feb 12, 2018 at 8:38 PM, ~$alpha` <[hidden email]> wrote:

> I cant test on more as performance is already degraded.
> Its a 32core system and load 25 means 2500% cpu
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
Yes, but how to move ahead  now.
Its strange solr4 is better behaving than solr6



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
If the community cannot help, the only way i can think is either to profile
Solr (java) under a load test to find the problem. You could also use an
APM.

On 12 Feb 2018 23:00, "~$alpha`" <[hidden email]> wrote:

> Yes, but how to move ahead  now.
> Its strange solr4 is better behaving than solr6
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Walter Underwood
In reply to this post by ~$alpha`
25 of 32 CPUs loaded is 78% CPU, but I’ve never seen CPU use reported that way.

How are you measuring CPU usage? What tool?

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Feb 12, 2018, at 7:08 AM, ~$alpha` <[hidden email]> wrote:
>
> I cant test on more as performance is already degraded.
> Its a 32core system and load 25 means 2500% cpu
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

Deepak Goel
In reply to this post by Deepak Goel
One more idea could be is to have multiple vm's (8 cpu each) on your server
and load balance them. That would help Solr6 scale nicely

On 12 Feb 2018 23:05, "Deepak Goel" <[hidden email]> wrote:

> If the community cannot help, the only way i can think is either to
> profile Solr (java) under a load test to find the problem. You could also
> use an APM.
>
> On 12 Feb 2018 23:00, "~$alpha`" <[hidden email]> wrote:
>
>> Yes, but how to move ahead  now.
>> Its strange solr4 is better behaving than solr6
>>
>>
>>
>> --
>> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr4 To Solr6 CPU load issues

~$alpha`
cpu utlization is already on higher end, so VM wont seems a solution



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
12