Random queries extremely slow

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Random queries extremely slow

oleg_gnatovskiy
Hello. Our production servers are operating relatively smoothly most of the time running Solr with 19 million listings. However every once in a while the same query that used to take 100 miliseconds takes 6000. This causes out health check to fail, and the server is taken out of service. Once the server is put back in service, queries are back to their regular response times. Is there anything we could do to stop this random slowness from occurring?
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

oleg_gnatovskiy
My aplogies, this is likely the same issue as "Intermittent high response times  by  hbi dev "


oleg_gnatovskiy wrote
Hello. Our production servers are operating relatively smoothly most of the time running Solr with 19 million listings. However every once in a while the same query that used to take 100 miliseconds takes 6000. This causes out health check to fail, and the server is taken out of service. Once the server is put back in service, queries are back to their regular response times. Is there anything we could do to stop this random slowness from occurring?
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

oleg_gnatovskiy
Actually my issue might merit a seperate discussion as I did tuning by adjusting the heap to different settings to see how it affected changed. It really had no affect, as with jdk 1.6, garbage collection is parallel which now should no longer interfere with requests during garbage collection which holds true based on the tests we ran.


oleg_gnatovskiy wrote
My aplogies, this is likely the same issue as "Intermittent high response times  by  hbi dev "


oleg_gnatovskiy wrote
Hello. Our production servers are operating relatively smoothly most of the time running Solr with 19 million listings. However every once in a while the same query that used to take 100 miliseconds takes 6000. This causes out health check to fail, and the server is taken out of service. Once the server is put back in service, queries are back to their regular response times. Is there anything we could do to stop this random slowness from occurring?
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

Yonik Seeley
In reply to this post by oleg_gnatovskiy
On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
<[hidden email]> wrote:
> Hello. Our production servers are operating relatively smoothly most of the
> time running Solr with 19 million listings. However every once in a while
> the same query that used to take 100 miliseconds takes 6000.

Anything else happening on the system that may have forced some of the
index files out of operating system disk cache at these times?

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

oleg_gnatovskiy
What are some things that could happen to force files out of the cache on a Linux machine? I don't know what kinds of events to look for...



yonik wrote
On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
<oleg_gnatovskiy@citysearch.com> wrote:
> Hello. Our production servers are operating relatively smoothly most of the
> time running Solr with 19 million listings. However every once in a while
> the same query that used to take 100 miliseconds takes 6000.

Anything else happening on the system that may have forced some of the
index files out of operating system disk cache at these times?

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

Walter Underwood, Netflix
The OS keeps recently accessed disk pages in memory. If another
process does a lot of disk access, like a backup, the OS might
replace the Solr index pages with that processes pages.

What kind of storage: local disk, SAN, NFS?

wunder

On 1/22/09 11:22 AM, "oleg_gnatovskiy" <[hidden email]>
wrote:

>
> What are some things that could happen to force files out of the cache on a
> Linux machine? I don't know what kinds of events to look for...
>
>
>
>
> yonik wrote:
>>
>> On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
>> <[hidden email]> wrote:
>>> Hello. Our production servers are operating relatively smoothly most of
>>> the
>>> time running Solr with 19 million listings. However every once in a while
>>> the same query that used to take 100 miliseconds takes 6000.
>>
>> Anything else happening on the system that may have forced some of the
>> index files out of operating system disk cache at these times?
>>
>> -Yonik
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

Otis Gospodnetic-2
In reply to this post by oleg_gnatovskiy
Here is one example: pushing a large newly optimized index onto the server.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----

> From: oleg_gnatovskiy <[hidden email]>
> To: [hidden email]
> Sent: Thursday, January 22, 2009 2:22:51 PM
> Subject: Re: Random queries extremely slow
>
>
> What are some things that could happen to force files out of the cache on a
> Linux machine? I don't know what kinds of events to look for...
>
>
>
>
> yonik wrote:
> >
> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> > wrote:
> >> Hello. Our production servers are operating relatively smoothly most of
> >> the
> >> time running Solr with 19 million listings. However every once in a while
> >> the same query that used to take 100 miliseconds takes 6000.
> >
> > Anything else happening on the system that may have forced some of the
> > index files out of operating system disk cache at these times?
> >
> > -Yonik
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

oleg_gnatovskiy
Well this probably isn't the cause of our random slow queries, but might be the cause of the slow queries after pulling a new index. Is there anything we could do to reduce the performance hit we take from this happening?


Otis Gospodnetic wrote
Here is one example: pushing a large newly optimized index onto the server.

Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch



----- Original Message ----
> From: oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>
> To: solr-user@lucene.apache.org
> Sent: Thursday, January 22, 2009 2:22:51 PM
> Subject: Re: Random queries extremely slow
>
>
> What are some things that could happen to force files out of the cache on a
> Linux machine? I don't know what kinds of events to look for...
>
>
>
>
> yonik wrote:
> >
> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> > wrote:
> >> Hello. Our production servers are operating relatively smoothly most of
> >> the
> >> time running Solr with 19 million listings. However every once in a while
> >> the same query that used to take 100 miliseconds takes 6000.
> >
> > Anything else happening on the system that may have forced some of the
> > index files out of operating system disk cache at these times?
> >
> > -Yonik
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> Sent from the Solr - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

Alexander Ramos Jardim
Use multiple boxes, with a mirroring delaay from one to another, like a
pipeline.

2009/1/22 oleg_gnatovskiy <[hidden email]>

>
> Well this probably isn't the cause of our random slow queries, but might be
> the cause of the slow queries after pulling a new index. Is there anything
> we could do to reduce the performance hit we take from this happening?
>
>
>
> Otis Gospodnetic wrote:
> >
> > Here is one example: pushing a large newly optimized index onto the
> > server.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy <[hidden email]>
> >> To: [hidden email]
> >> Sent: Thursday, January 22, 2009 2:22:51 PM
> >> Subject: Re: Random queries extremely slow
> >>
> >>
> >> What are some things that could happen to force files out of the cache
> on
> >> a
> >> Linux machine? I don't know what kinds of events to look for...
> >>
> >>
> >>
> >>
> >> yonik wrote:
> >> >
> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> >> > wrote:
> >> >> Hello. Our production servers are operating relatively smoothly most
> >> of
> >> >> the
> >> >> time running Solr with 19 million listings. However every once in a
> >> while
> >> >> the same query that used to take 100 miliseconds takes 6000.
> >> >
> >> > Anything else happening on the system that may have forced some of the
> >> > index files out of operating system disk cache at these times?
> >> >
> >> > -Yonik
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


--
Alexander Ramos Jardim
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

oleg_gnatovskiy
Can you expand on this? Mirroring delay on what?


zayhen wrote
Use multiple boxes, with a mirroring delaay from one to another, like a
pipeline.

2009/1/22 oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>

>
> Well this probably isn't the cause of our random slow queries, but might be
> the cause of the slow queries after pulling a new index. Is there anything
> we could do to reduce the performance hit we take from this happening?
>
>
>
> Otis Gospodnetic wrote:
> >
> > Here is one example: pushing a large newly optimized index onto the
> > server.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, January 22, 2009 2:22:51 PM
> >> Subject: Re: Random queries extremely slow
> >>
> >>
> >> What are some things that could happen to force files out of the cache
> on
> >> a
> >> Linux machine? I don't know what kinds of events to look for...
> >>
> >>
> >>
> >>
> >> yonik wrote:
> >> >
> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> >> > wrote:
> >> >> Hello. Our production servers are operating relatively smoothly most
> >> of
> >> >> the
> >> >> time running Solr with 19 million listings. However every once in a
> >> while
> >> >> the same query that used to take 100 miliseconds takes 6000.
> >> >
> >> > Anything else happening on the system that may have forced some of the
> >> > index files out of operating system disk cache at these times?
> >> >
> >> > -Yonik
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


--
Alexander Ramos Jardim


-----
RPG da Ilha
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

sunnyShiny06
Hi Oleg

Did you find a way to pass over this issue ??
thanks a lot,

oleg_gnatovskiy wrote
Can you expand on this? Mirroring delay on what?


zayhen wrote
Use multiple boxes, with a mirroring delaay from one to another, like a
pipeline.

2009/1/22 oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>

>
> Well this probably isn't the cause of our random slow queries, but might be
> the cause of the slow queries after pulling a new index. Is there anything
> we could do to reduce the performance hit we take from this happening?
>
>
>
> Otis Gospodnetic wrote:
> >
> > Here is one example: pushing a large newly optimized index onto the
> > server.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, January 22, 2009 2:22:51 PM
> >> Subject: Re: Random queries extremely slow
> >>
> >>
> >> What are some things that could happen to force files out of the cache
> on
> >> a
> >> Linux machine? I don't know what kinds of events to look for...
> >>
> >>
> >>
> >>
> >> yonik wrote:
> >> >
> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> >> > wrote:
> >> >> Hello. Our production servers are operating relatively smoothly most
> >> of
> >> >> the
> >> >> time running Solr with 19 million listings. However every once in a
> >> while
> >> >> the same query that used to take 100 miliseconds takes 6000.
> >> >
> >> > Anything else happening on the system that may have forced some of the
> >> > index files out of operating system disk cache at these times?
> >> >
> >> > -Yonik
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


--
Alexander Ramos Jardim


-----
RPG da Ilha
Reply | Threaded
Open this post in threaded view
|

Re: Random queries extremely slow

oleg_gnatovskiy
It was actually our use of the field collapse patch. Once we disabled this the random slow queries went away.

We also added *:* as a warmup query in order to speed up performance after indexing.


sunnyfr wrote
Hi Oleg

Did you find a way to pass over this issue ??
thanks a lot,

oleg_gnatovskiy wrote
Can you expand on this? Mirroring delay on what?


zayhen wrote
Use multiple boxes, with a mirroring delaay from one to another, like a
pipeline.

2009/1/22 oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>

>
> Well this probably isn't the cause of our random slow queries, but might be
> the cause of the slow queries after pulling a new index. Is there anything
> we could do to reduce the performance hit we take from this happening?
>
>
>
> Otis Gospodnetic wrote:
> >
> > Here is one example: pushing a large newly optimized index onto the
> > server.
> >
> > Otis
> > --
> > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch
> >
> >
> >
> > ----- Original Message ----
> >> From: oleg_gnatovskiy <oleg_gnatovskiy@citysearch.com>
> >> To: solr-user@lucene.apache.org
> >> Sent: Thursday, January 22, 2009 2:22:51 PM
> >> Subject: Re: Random queries extremely slow
> >>
> >>
> >> What are some things that could happen to force files out of the cache
> on
> >> a
> >> Linux machine? I don't know what kinds of events to look for...
> >>
> >>
> >>
> >>
> >> yonik wrote:
> >> >
> >> > On Thu, Jan 22, 2009 at 1:46 PM, oleg_gnatovskiy
> >> > wrote:
> >> >> Hello. Our production servers are operating relatively smoothly most
> >> of
> >> >> the
> >> >> time running Solr with 19 million listings. However every once in a
> >> while
> >> >> the same query that used to take 100 miliseconds takes 6000.
> >> >
> >> > Anything else happening on the system that may have forced some of the
> >> > index files out of operating system disk cache at these times?
> >> >
> >> > -Yonik
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611240.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Random-queries-extremely-slow-tp21610568p21611454.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
>


--
Alexander Ramos Jardim


-----
RPG da Ilha