abnormal cpu usage (solr 7.3.1)

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

abnormal cpu usage (solr 7.3.1)

Danilo Tomasoni
Hello all,

we have a solr instance with around 41MLN documents.

Recently we stopped our forcemerge policy that ensured only 1 segment
was present at query time, because we read here in the ML and elsewhere
that this is considered bad practice and it's also very time consuming.

So we implemented

<mergePolicyFactory class="org.apache.lucene.index.LogByteSizeMergePolicy">
   <int name="mergeFactor">10</int>
</mergePolicyFactory>

I don't know if this is related to our problem, but recently we noticed
spikes in CPU usage (without iowait, and that's the strange thing,
because otherwise I would have thought about an automatic merge)

every 3-5 seconds.

I attach a picture of the metrics collected from the system hosting solr
that show this behaviour, as well as the gc log.

Any idea why this can happen?

Thank you

Danilo

--
Danilo Tomasoni

Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI)
Piazza Manifattura 1,  38068 Rovereto (TN), Italy
[hidden email]
http://www.cosbi.eu
 
As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to


solr_gc.log.7.current.bz2 (253K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] abnormal cpu usage (solr 7.3.1)

Danilo Tomasoni
I'm sorry I forgot the pictures.

On 11/12/19 15:20, Danilo Tomasoni wrote:

> Hello all,
>
> we have a solr instance with around 41MLN documents.
>
> Recently we stopped our forcemerge policy that ensured only 1 segment
> was present at query time, because we read here in the ML and
> elsewhere that this is considered bad practice and it's also very time
> consuming.
>
> So we implemented
>
> <mergePolicyFactory
> class="org.apache.lucene.index.LogByteSizeMergePolicy">
>   <int name="mergeFactor">10</int>
> </mergePolicyFactory>
>
> I don't know if this is related to our problem, but recently we
> noticed spikes in CPU usage (without iowait, and that's the strange
> thing, because otherwise I would have thought about an automatic merge)
>
> every 3-5 seconds.
>
> I attach a picture of the metrics collected from the system hosting
> solr that show this behaviour, as well as the gc log.
>
> Any idea why this can happen?
>
> Thank you
>
> Danilo
>
--
Danilo Tomasoni

Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI)
Piazza Manifattura 1,  38068 Rovereto (TN), Italy
[hidden email]
http://www.cosbi.eu
 
As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR.
It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
P Please don't print this e-mail unless you really need to

Reply | Threaded
Open this post in threaded view
|

Re: [SPAM] abnormal cpu usage (solr 7.3.1)

Erick Erickson
Why are you using LogByteSIzeMergePolicy? The current default is TieredMergePolicy, unless you have a specific reason, I’d just take explicit merge policies out of your solrconfig and use the defaults.

That aside,  I doubt it’s related. You could well see these spikes due to background merging, but that should have been present before as well.

Every 3-5 seconds is interesting, does that coincide with commits and/or opening searchers? If opening searchers, quite possibly autowarming if you have that configured.

> On Dec 11, 2019, at 9:26 AM, Danilo Tomasoni <[hidden email]> wrote:
>
> I'm sorry I forgot the pictures.
>
> On 11/12/19 15:20, Danilo Tomasoni wrote:
>> Hello all,
>>
>> we have a solr instance with around 41MLN documents.
>>
>> Recently we stopped our forcemerge policy that ensured only 1 segment was present at query time, because we read here in the ML and elsewhere that this is considered bad practice and it's also very time consuming.
>>
>> So we implemented
>>
>> <mergePolicyFactory class="org.apache.lucene.index.LogByteSizeMergePolicy">
>>   <int name="mergeFactor">10</int>
>> </mergePolicyFactory>
>>
>> I don't know if this is related to our problem, but recently we noticed spikes in CPU usage (without iowait, and that's the strange thing, because otherwise I would have thought about an automatic merge)
>>
>> every 3-5 seconds.
>>
>> I attach a picture of the metrics collected from the system hosting solr that show this behaviour, as well as the gc log.
>>
>> Any idea why this can happen?
>>
>> Thank you
>>
>> Danilo
>>
> --
> Danilo Tomasoni
>
> Fondazione The Microsoft Research - University of Trento Centre for Computational and Systems Biology (COSBI)
> Piazza Manifattura 1,  38068 Rovereto (TN), Italy
> [hidden email]
> http://www.cosbi.eu
> As for the European General Data Protection Regulation 2016/679 on the protection of natural persons with regard to the processing of personal data, we inform you that all the data we possess are object of treatment in the respect of the normative provided for by the cited GDPR.
> It is your right to be informed on which of your data are used and how; you may ask for their correction, cancellation or you may oppose to their use by written request sent by recorded delivery to The Microsoft Research – University of Trento Centre for Computational and Systems Biology Scarl, Piazza Manifattura 1, 38068 Rovereto (TN), Italy.
> P Please don't print this e-mail unless you really need to
>