replication caching high query and lot of update

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

replication caching high query and lot of update

sunnyShiny06
Hi everybody,

This is my issue :
I've a master which update 20 000 docs every 30mn. (and a lot more nightly)
So my index is merging almost every update, segment increase too much.
So my master replicate all the index to the slave almost every 30mn.

My point is, my slaves are very slow during it get back index by replication script.
I enclosed my cpu activity just during an update. CPU.jpg
Then you can Imagine 15-20 request second turnd badly my cpu.

>What is the best configuration for tomcat with this kind of activity?

I've linux / solr 1.4 / 8G RAM  / 8CPU.
Data index size : 11G - 14Mdocs

>And about my cache, with a such activity, is it interesting to have a cache stored or not ??

My big point is during replication, my respond time of my request is sooo slow.

Thanks a lot,


Reply | Threaded
Open this post in threaded view
|

Re: replication caching high query and lot of update

Ted Dunning
This may be largely due to poor I/O scheduling at the OS layer.

Try switching to an I/O scheduler that puts reads ahead of writes.

On Wed, Apr 1, 2009 at 8:20 AM, sunnyfr <[hidden email]> wrote:

> >And about my cache, with a such activity, is it interesting to have a
> cache
> stored or not ??
>
> My big point is during replication, my respond time of my request is sooo
> slow.
>



--
Ted Dunning, CTO
DeepDyve
Reply | Threaded
Open this post in threaded view
|

Re: replication caching high query and lot of update

sunnyShiny06
Hi Ted,

Do you have advice for doing that ???
I've linux.

Just another question how many % of memory would you give to the jvm.
I've 8G of ram (8cpu) and my index data is 11G. what would you reckon as xmx?


Thanks a lot,

Ted Dunning wrote
This may be largely due to poor I/O scheduling at the OS layer.

Try switching to an I/O scheduler that puts reads ahead of writes.

On Wed, Apr 1, 2009 at 8:20 AM, sunnyfr <johanna.34@gmail.com> wrote:

> >And about my cache, with a such activity, is it interesting to have a
> cache
> stored or not ??
>
> My big point is during replication, my respond time of my request is sooo
> slow.
>



--
Ted Dunning, CTO
DeepDyve
Reply | Threaded
Open this post in threaded view
|

Re: replication caching high query and lot of update

Yonik Seeley-2-2
On Thu, Apr 2, 2009 at 7:29 AM, sunnyfr <[hidden email]> wrote:
> Just another question how many % of memory would you give to the jvm.
> I've 8G of ram (8cpu) and my index data is 11G. what would you reckon as
> xmx?

This is really Solr specific stuff and should be on solr-user.

You want to give the JVM the least amount of memory such that
everything still works, and allowing for a little index growth.  What
that amount is will depend a lot on which fields you sort on , which
fields you facet on, all your caches, etc.  The reason you want to
minimize the amount of JVM memory is so that the OS can cache
important parts of the index with the remaining free RAM.

Large heap sizes also lead to long GC pauses.


-Yonik
http://www.lucidimagination.com