Smaller filterCache giving better performance

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Smaller filterCache giving better performance

wojtekpia
I've seen some strangle results in the last few days of testing, but this one flies in the face of everything I've read on this forum: Reducing filterCache size has increased performance.

I have posted my setup here: http://www.nabble.com/Throughput-Optimization-td20335132.html.

My original filterCache was 700,000. Reducing it to 20,000, I found:
- Average response time decreased by 85%
- Average throughput increased by 250%
- CPU time used by the garbage collector decreased by 85%
- The system showed no weird GC issues (reported yesterday at: http://www.nabble.com/new-faceting-algorithm-td20674902.html)

Further reducing the filterCache to 10,000
- Average response time decreased by another 27%
- Average throughput increased by another 30%
- GC CPU usage also dropped
- System behavior changed after ~30 minutes, with a slight performance degradation

These results came from a load test. I'm running trunk code from Dec 2 with Yonik's faceting improvement turned on.

Any thoughts?
Reply | Threaded
Open this post in threaded view
|

Re: Smaller filterCache giving better performance

Mike Klaas
On 5-Dec-08, at 2:24 PM, wojtekpia wrote:

>
> I've seen some strangle results in the last few days of testing, but  
> this one
> flies in the face of everything I've read on this forum: Reducing
> filterCache size has increased performance.

This isn't really unexpected behaviour.  The problem with a huge  
filter cache is that it is fighting with OS disk cache--the latter of  
which can be much much more important.  Reducing the size of the  
filter cache give more to the OS.

Try giving 17GB to java, and letting the OS cache the entire index.  
Increase the filter cache as much as you can without OOM'ing.  That  
should give optimal performance.   Note that you don't always need the  
_whole_ index in the os cache to get acceptable performance, but if  
you can afford it, it is a good idea.

It is also possible that you are experiencing contention in the  
filtercache code--have you tried the concurrent filter cache impl?

-Mike

>
> I have posted my setup here:
> http://www.nabble.com/Throughput-Optimization-td20335132.html.
>
> My original filterCache was 700,000. Reducing it to 20,000, I found:
> - Average response time decreased by 85%
> - Average throughput increased by 250%
> - CPU time used by the garbage collector decreased by 85%
> - The system showed to weird GC issues (reported yesterday at:
> http://www.nabble.com/new-faceting-algorithm-td20674902.html)
>
> Further reducing the filterCache to 10,000
> - Average response time decreased by another 27%
> - Average throughput increased by another 30%
> - GC CPU usage also dropped
> - System behavior changed after ~30 minutes, with a slight performance
> degradation
>
> These results came from a load test. I'm running trunk code from Dec  
> 2 with
> Yonik's faceting improvement turned on.
>
> Any thoughts?
> --
> View this message in context: http://www.nabble.com/Smaller-filterCache-giving-better-performance-tp20863674p20863674.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Smaller filterCache giving better performance

Yonik Seeley-2
In reply to this post by wojtekpia
On Fri, Dec 5, 2008 at 5:24 PM, wojtekpia <[hidden email]> wrote:

>
> I've seen some strangle results in the last few days of testing, but this one
> flies in the face of everything I've read on this forum: Reducing
> filterCache size has increased performance.
>
> I have posted my setup here:
> http://www.nabble.com/Throughput-Optimization-td20335132.html.
>
> My original filterCache was 700,000. Reducing it to 20,000, I found:
> - Average response time decreased by 85%
> - Average throughput increased by 250%
> - CPU time used by the garbage collector decreased by 85%
> - The system showed to weird GC issues (reported yesterday at:
> http://www.nabble.com/new-faceting-algorithm-td20674902.html)
>
> Further reducing the filterCache to 10,000
> - Average response time decreased by another 27%
> - Average throughput increased by another 30%
> - GC CPU usage also dropped
> - System behavior changed after ~30 minutes, with a slight performance
> degradation
>
> These results came from a load test. I'm running trunk code from Dec 2 with
> Yonik's faceting improvement turned on.

Old faceting used the filterCache exclusively.
New faceting only uses it for terms that cover ~5% of the index, so
you can reduce the filterCache quite a bit potentially, save more RAM,
and increase the amount of memory you can give to the OS cache.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Smaller filterCache giving better performance

wojtekpia
In reply to this post by Mike Klaas
Reducing the amount of memory given to java slowed down Solr at first, then quickly caused the garbage collector to behave badly (same issue as I referenced above).

I am using the concurrent cache for all my caches.