Autocommit

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Autocommit

Michael Thessel
Hello UG

I already posted a while ago a problem that one of the solr threads
starts using 100% of one of the processor cores on a 4 core
system. This doesn't happen right after the start it slightly
increaes for about a week until the process runs constantly at
100%. I couldn't figure out a solution for this. I could live with this
problem but I think it has an side effect. While the processor load
increases the time between two autocommits increases as well. Currently
autocommit is set to 3 minutes. After 4 weeks the commits run only
every 40 minutes.

I have the following solr version installed:

Solr Specification Version: 1.2.0
Solr Implementation Version: 1.2.0 - Yonik - 2007-06-02 17:35:12
Lucene Specification Version: 2007-05-20_00-04-53
Lucene Implementation Version: build 2007-05-20

Does anyone have a hint what I could look for?

Thanks

Michael

--
Michael Thessel <[hidden email]>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806
Reply | Threaded
Open this post in threaded view
|

Re: Autocommit

Yonik Seeley-2
On Dec 12, 2007 6:15 PM, Michael Thessel <[hidden email]> wrote:
> I already posted a while ago a problem that one of the solr threads
> starts using 100% of one of the processor cores on a 4 core
> system.

This sounds like warming / autowarming.
The other possibility is garbage collection.

> This doesn't happen right after the start it slightly
> increaes for about a week until the process runs constantly at
> 100%.

Is it still just one CPU at 100%, or is it ever 2 or more at 100%.
That would tell us if it were due to overlapping autowarming.

What happens to your index over time?  Does maxDoc() keep increasing?

> I couldn't figure out a solution for this. I could live with this
> problem but I think it has an side effect. While the processor load
> increases the time between two autocommits increases as well. Currently
> autocommit is set to 3 minutes. After 4 weeks the commits run only
> every 40 minutes.
>
> I have the following solr version installed:
>
> Solr Specification Version: 1.2.0
> Solr Implementation Version: 1.2.0 - Yonik - 2007-06-02 17:35:12
> Lucene Specification Version: 2007-05-20_00-04-53
> Lucene Implementation Version: build 2007-05-20
>
> Does anyone have a hint what I could look for?

Perhaps post the XML you get from the statistics page so we might know more.
Try looking in the logs to see what part of autowarming is taking so long.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Autocommit

Michael Thessel
> > I already posted a while ago a problem that one of the solr threads
> > starts using 100% of one of the processor cores on a 4 core
> > system.
>
> This sounds like warming / autowarming.
> The other possibility is garbage collection.

What can I do here? Decrease the autowarmcount?

My current settings
filterCache autowarmCount="256"
queryResultCache autowarmCount="256"
documentCache autowarmCount="0"

> > This doesn't happen right after the start it slightly
> > increaes for about a week until the process runs constantly at
> > 100%.
>
> Is it still just one CPU at 100%, or is it ever 2 or more at 100%.
> That would tell us if it were due to overlapping autowarming.

It is always just one core. All the other cores run once in a while at
100% but only one core is constantly at 100%

> What happens to your index over time?  Does maxDoc() keep increasing?
Yes maxDoc is always increasing it is pretty much the same as the total
number of indexed documents.

> Perhaps post the XML you get from the statistics page so we might
> know more. Try looking in the logs to see what part of autowarming is
> taking so long.
While checking the logs I run over some performance warnings. I run a
optimize every night. Minutes after I start the optimize I get a:

INFO: PERFORMANCE WARNING: Overlapping onDeckSearchers=2

Could the problem be related to this? I disabled the optimize for now.
If the optimize is the problem what would be a better different
strategy to run the optimize?

Thanks for your help.

Cheers,

Michael

--
Michael Thessel <[hidden email]>
Gossamer Threads Inc. http://www.gossamer-threads.com/
Tel: (604) 687-5804 Fax: (604) 687-5806