Slow Response for less volume

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow Response for less volume

Amjad Khan
Hi,

We recently moved to Solr Cloud (Google) with 4 nodes and have very limited number of data.

We are facing very weird issue here, solr cluster response time for query is high when we have less number of hit and the moment we run our test to hit the solr cluster hard we see better response in 10ms.

Any clue will be appreciated.

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Walter Underwood
Are you testing with a small number of queries? If your cache is larger than the number of queries in your benchmark, the first round will load the cache, then everything will be super fast.

Load testing a system with caches is hard.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Oct 24, 2018, at 9:19 AM, Amjad Khan <[hidden email]> wrote:
>
> Hi,
>
> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited number of data.
>
> We are facing very weird issue here, solr cluster response time for query is high when we have less number of hit and the moment we run our test to hit the solr cluster hard we see better response in 10ms.
>
> Any clue will be appreciated.
>
> Thanks

Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Erick Erickson
You can set your queryResultCache and filterCache "size" parameter to
zero in solrconfig.xml to disable those caches.
On Wed, Oct 24, 2018 at 9:21 AM Walter Underwood <[hidden email]> wrote:

>
> Are you testing with a small number of queries? If your cache is larger than the number of queries in your benchmark, the first round will load the cache, then everything will be super fast.
>
> Load testing a system with caches is hard.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
> > On Oct 24, 2018, at 9:19 AM, Amjad Khan <[hidden email]> wrote:
> >
> > Hi,
> >
> > We recently moved to Solr Cloud (Google) with 4 nodes and have very limited number of data.
> >
> > We are facing very weird issue here, solr cluster response time for query is high when we have less number of hit and the moment we run our test to hit the solr cluster hard we see better response in 10ms.
> >
> > Any clue will be appreciated.
> >
> > Thanks
>
Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Amjad Khan
In reply to this post by Walter Underwood
Thanks Wunder for this prompt response.

We are testing with 1600 different text to search with Jmeter and that keeps running continuously, and keep running continuously means cache has been built and there should be better response now. Doesn’t it?

Thanks



> On Oct 24, 2018, at 12:20 PM, Walter Underwood <[hidden email]> wrote:
>
> Are you testing with a small number of queries? If your cache is larger than the number of queries in your benchmark, the first round will load the cache, then everything will be super fast.
>
> Load testing a system with caches is hard.
>
> wunder
> Walter Underwood
> [hidden email]
> http://observer.wunderwood.org/  (my blog)
>
>> On Oct 24, 2018, at 9:19 AM, Amjad Khan <[hidden email]> wrote:
>>
>> Hi,
>>
>> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited number of data.
>>
>> We are facing very weird issue here, solr cluster response time for query is high when we have less number of hit and the moment we run our test to hit the solr cluster hard we see better response in 10ms.
>>
>> Any clue will be appreciated.
>>
>> Thanks
>

Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Amjad Khan
In reply to this post by Erick Erickson
Thanks Erick,

But do you think that disabling the cache will increase the response time instead of solving the problem here.


> On Oct 24, 2018, at 12:52 PM, Erick Erickson <[hidden email]> wrote:
>
> queryResultCache

Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Walter Underwood
In reply to this post by Erick Erickson
But a zero size cache doesn’t give realistic benchmarks. It makes things slower than they will be in production.

We do this:

1. Collect production logs.
2. Split the logs into a warming log and and a benchmark log. The warming log should be at least as large as the query result cache.
3. Run the warming log with four threads (unlikely to overload the system).
4. Run the benchmark with a controlled requests/minute and enough threads to keep up with that. Might be a few hundred with a large, slow cluster. Run for at least an hour.
5. Analyze the results into percentile response times for each request handler. Warn about any errors or a benchmark that takes too long.

Then reepeat. Oh, yeah, load the prod content first.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Oct 24, 2018, at 9:52 AM, Erick Erickson <[hidden email]> wrote:
>
> You can set your queryResultCache and filterCache "size" parameter to
> zero in solrconfig.xml to disable those caches.
> On Wed, Oct 24, 2018 at 9:21 AM Walter Underwood <[hidden email]> wrote:
>>
>> Are you testing with a small number of queries? If your cache is larger than the number of queries in your benchmark, the first round will load the cache, then everything will be super fast.
>>
>> Load testing a system with caches is hard.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Oct 24, 2018, at 9:19 AM, Amjad Khan <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited number of data.
>>>
>>> We are facing very weird issue here, solr cluster response time for query is high when we have less number of hit and the moment we run our test to hit the solr cluster hard we see better response in 10ms.
>>>
>>> Any clue will be appreciated.
>>>
>>> Thanks
>>

Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Walter Underwood
In reply to this post by Amjad Khan
If your cache is 2048 entries, then every one of those 1600 queries is in cache.

Our logs typically have about a million lines, with distinct queries distributed according to the Zipf law. Some common queries, a long tail, that sort of thing.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)

> On Oct 24, 2018, at 10:02 AM, Amjad Khan <[hidden email]> wrote:
>
> Thanks Wunder for this prompt response.
>
> We are testing with 1600 different text to search with Jmeter and that keeps running continuously, and keep running continuously means cache has been built and there should be better response now. Doesn’t it?
>
> Thanks
>
>
>
>> On Oct 24, 2018, at 12:20 PM, Walter Underwood <[hidden email]> wrote:
>>
>> Are you testing with a small number of queries? If your cache is larger than the number of queries in your benchmark, the first round will load the cache, then everything will be super fast.
>>
>> Load testing a system with caches is hard.
>>
>> wunder
>> Walter Underwood
>> [hidden email]
>> http://observer.wunderwood.org/  (my blog)
>>
>>> On Oct 24, 2018, at 9:19 AM, Amjad Khan <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> We recently moved to Solr Cloud (Google) with 4 nodes and have very limited number of data.
>>>
>>> We are facing very weird issue here, solr cluster response time for query is high when we have less number of hit and the moment we run our test to hit the solr cluster hard we see better response in 10ms.
>>>
>>> Any clue will be appreciated.
>>>
>>> Thanks
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: Slow Response for less volume

Deepak Goel
In reply to this post by Amjad Khan
Are you getting errors in Jmeter?

On Wed, 24 Oct 2018, 21:49 Amjad Khan, <[hidden email]> wrote:

> Hi,
>
> We recently moved to Solr Cloud (Google) with 4 nodes and have very
> limited number of data.
>
> We are facing very weird issue here, solr cluster response time for query
> is high when we have less number of hit and the moment we run our test to
> hit the solr cluster hard we see better response in 10ms.
>
> Any clue will be appreciated.
>
> Thanks