Major CPU performance problems under heavy user load with solr 1.2

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Major CPU performance problems under heavy user load with solr 1.2

Robert Purdy
Hi there, I am having some major CPU performance problems with heavy user load with solr 1.2. I currently have approximately 4 million documents in the index and I am doing some pretty heavy faceting on multi-valued columns. I know that doing facets are expensive on multi-valued columns but the CPU seems to max out (400%) with apache bench with just 5 identical concurrent requests and I have the potential for a lot more concurrent requests then that with my large number of users that hit our site per day and I am wondering if there are any workarounds. Currently I am running the out of the box solr solution (Example jetty application with my own schema.xml and solrconfig.xml) on a dual Intel Duo core 64 bit box with 8 gigs of ram allocated to the start.jar process dedicated to solr with no slaves.

I have set up some aggressive caching in the solrconfig.xml for the filtercache (class="solr.LRUCache"size="3000000" initialSize="2000000") and have the HashDocSet set to 10000 to help with faceting, but still I am getting some pretty poor performance. I have also tried autowarming the facets by performing a query that hits all my multivalued facets with no facet limits across all the documents in the index. This does seem to reduce my query times by a lot because the filtercache grows to about 2.1 million lookups and finishes the query in about 70 secs. However I have noticed an issue with this because each time I do an optimize or a commit after prewarming the facets the cache gets cleared, according to the stats on the admin page, but the RSize does not shink for the process, and the queries get slow again, so I prewarm the facets again and the memory usage keeps growing like the cache is not being recycled and as a results the prewarm query starts to get slower and slower as each time this occurs (after about 5 times of prewarms and then commit the query takes about 30 mins... ugh) and almost run out of memory.

Any thoughts on how to help improve this and fix the memory issue?
Reply | Threaded
Open this post in threaded view
|

Re: Major CPU performance problems under heavy user load with solr 1.2

Yonik Seeley-2
On 10/1/07, Robert Purdy <[hidden email]> wrote:
> Hi there, I am having some major CPU performance problems with heavy user
> load with solr 1.2. I currently have approximately 4 million documents in
> the index and I am doing some pretty heavy faceting on multi-valued columns.
> I know that doing facets are expensive on multi-valued columns but the CPU
> seems to max out (400%) with apache bench with just 5 identical concurrent
> requests

One can always max out CPU (unless one is IO bound) with concurrent
requests greater than the number of CPUs on the system.  This isn't a
problem by itself and would exist even if Solr were an order of
magnitude slower or faster.

You should be looking at things the peak throughput (queries per sec)
you need to support and the latency of the requests (look at the 90
percentile, or whatever).


> and I have the potential for a lot more concurrent requests then
> that with my large number of users that hit our site per day and I am
> wondering if there are any workarounds. Currently I am running the out of
> the box solr solution (Example jetty application with my own schema.xml and
> solrconfig.xml) on a dual Intel Duo core 64 bit box with 8 gigs of ram
> allocated to the start.jar process dedicated to solr with no slaves.
>
> I have set up some aggressive caching in the solrconfig.xml for the
> filtercache (class="solr.LRUCache"size="3000000" initialSize="2000000") and
> have the HashDocSet set to 10000 to help with faceting, but still I am
> getting some pretty poor performance. I have also tried autowarming the
> facets by performing a query that hits all my multivalued facets with no
> facet limits across all the documents in the index. This does seem to reduce
> my query times by a lot because the filtercache grows to about 2.1 million
> lookups and finishes the query in about 70 secs.

OK, that's long.  So focus on the latency of a single request instead
of jumping straight to load testing.

2.1 million is a lot - what's the field with the largest number of
unique values that you are faceting on?

> However I have noticed an
> issue with this because each time I do an optimize or a commit after
> prewarming the facets the cache gets cleared, according to the stats on the
> admin page, but the RSize does not shink for the process, and the queries
> get slow again, so I prewarm the facets again and the memory usage keeps
> growing like the cache is not being recycled

The old searcher and cache won't be discarded until all requests using
it have completed.

> and as a results the prewarm
> query starts to get slower and slower as each time this occurs (after about
> 5 times of prewarms and then commit the query takes about 30 mins... ugh)
> and almost run out of memory.
>
> Any thoughts on how to help improve this and fix the memory issue?

You could try the minDf param to reduce the number of facets stored in
the cache and reduce memory consumption.

-Yonik