Performance tuning

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Performance tuning

Stephanie Belton
Hello,

 

Solr is now up and running on our production environment and working great. However it is taking up a lot of extra CPU and memory (CPU usage has doubled and memory is swapping). Is there any documentation on performance tuning? There seems to be a lot of useful info in the server output but I don’t understand it.

 

E.g.        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=537,evictions=0,size=337,cumulative_lookups=4723,cumulative_hits=3708,cumulative_hitratio=0.78,cumulative_inserts=4647,cumulative_evictions=72}

        queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=256,evictions=0,size=256,cumulative_lookups=3779,cumulative_hits=552,cumulative_hitratio=0.14,cumulative_inserts=3632,cumulative_evictions=0}

        documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=66005,cumulative_hits=2460,cumulative_hitratio=0.03,cumulative_inserts=63545,cumulative_evictions=4195}

 

etc. what should I be watching out for?

 

Thanks

Stephanie

Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Thorsten Scherler-3
On Thu, 2007-01-11 at 14:57 +0000, Stephanie Belton wrote:

> Hello,
>
>  
>
> Solr is now up and running on our production environment and working great. However it is taking up a lot of extra CPU and memory (CPU usage has doubled and memory is swapping). Is there any documentation on performance tuning? There seems to be a lot of useful info in the server output but I don’t understand it.
>
>  
>
> E.g.        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=537,evictions=0,size=337,cumulative_lookups=4723,cumulative_hits=3708,cumulative_hitratio=0.78,cumulative_inserts=4647,cumulative_evictions=72}
>
>         queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=256,evictions=0,size=256,cumulative_lookups=3779,cumulative_hits=552,cumulative_hitratio=0.14,cumulative_inserts=3632,cumulative_evictions=0}
>
>         documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=66005,cumulative_hits=2460,cumulative_hitratio=0.03,cumulative_inserts=63545,cumulative_evictions=4195}
>
>  
>
> etc. what should I be watching out for?
>

Hi Stephanie,

did you see http://wiki.apache.org/solr/SolrPerformanceFactors?

Further you may consider to balance the load via
http://wiki.apache.org/solr/CollectionDistribution

HTH

salu2

>  
>
> Thanks
>
> Stephanie
>

Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Yonik Seeley-2
In reply to this post by Stephanie Belton
On 1/11/07, Stephanie Belton <[hidden email]> wrote:

> Solr is now up and running on our production environment and working great. However it is taking up a lot of extra CPU and memory (CPU usage has doubled and memory is swapping). Is there any documentation on performance tuning? There seems to be a lot of useful info in the server output but I don't understand it.

Swapping if it's constant isn't good...  How much memory does this box
have, and what is the heap size of the JVM?  Are there other things
running on this box?

Solr does warming of caches by default to make complex queries that
hit a new snapshot of the index fast.  This takes up CPU in bursts,
but is normally nothing to worry about unless you have other apps
running on the same box that need CPU.  Because of this warming, CPU
usage of a Solr collection isn't directly related to query traffic at
all times.


-Yonik
Reply | Threaded
Open this post in threaded view
|

RE: Performance tuning

Stephanie Belton-2
This is the output of the free command:

[root@eul0000846 root2]# free -m
             total       used       free     shared    buffers     cached
Mem:          2007       1888        119          0         86        814
-/+ buffers/cache:        986       1020
Swap:         1992        207       1784

We normally have no swapping at all on this server and since last night
(when Solr was deployed on the site) it's been going up.

Here is an extract of the top command output sorted by memory usage, does
each of the processes really take up 566M??? CU usage is low because we are
outside of peak time but during the day it's at 40% when it used to be just
20%:

20:14:16  up 45 days, 21:47,  1 user,  load average: 1.06, 1.14, 1.11
167 processes: 166 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    8.8%    0.0%    0.3%   0.1%     0.2%    6.9%   83.2%
           cpu00    7.9%    0.0%    0.3%   0.7%     0.9%    6.9%   82.8%
           cpu01    8.5%    0.0%    0.3%   0.0%     0.0%    6.9%   84.0%
           cpu02    9.9%    0.0%    0.1%   0.0%     0.0%    6.9%   82.8%
           cpu03    9.0%    0.0%    0.6%   0.0%     0.2%    7.0%   83.2%
Mem:  2055300k av, 1914588k used,  140712k free,       0k shrd,   89032k
buff
                   1326540k actv,  301236k in_d,   30788k in_c
Swap: 2040244k av,  212948k used, 1827296k free                  843380k
cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
12201 root      15   0  566M 561M 13276 S     0.0 27.9   0:02   0 java
12203 root      15   0  566M 561M 13276 S     0.0 27.9   4:48   2 java
12204 root      16   0  566M 561M 13276 S     0.0 27.9   4:45   1 java
12205 root      15   0  566M 561M 13276 S     0.0 27.9   4:45   0 java
12206 root      15   0  566M 561M 13276 S     0.0 27.9   4:46   2 java
12207 root      15   0  566M 561M 13276 S     0.0 27.9   8:35   2 java
12208 root      16   0  566M 561M 13276 S     0.0 27.9  15:53   1 java
12209 root      16   0  566M 561M 13276 S     0.0 27.9  27:30   1 java
12210 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
12211 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
12212 root      15   0  566M 561M 13276 S     0.0 27.9   0:17   1 java
12213 root      15   0  566M 561M 13276 S     0.0 27.9   0:15   2 java
12214 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
12215 root      15   0  566M 561M 13276 S     0.0 27.9   0:33   2 java
12217 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
12218 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
12219 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
12220 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
12221 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
12222 root      25   0  566M 561M 13276 S     0.0 27.9 297:21   2 java
12223 root      15   0  566M 561M 13276 S     0.0 27.9   0:13   3 java
12224 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
12225 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
12226 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
12227 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
12228 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
12229 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
12230 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
Etc...

On the server we also have a website running using mod_perl, it's been
running for 1 year and up until now the CPU usage was peaking at 20% and
memory around 28% no swapping.

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: 11 January 2007 15:12
To: [hidden email]
Subject: Re: Performance tuning

On 1/11/07, Stephanie Belton <[hidden email]> wrote:

> Solr is now up and running on our production environment and working
great. However it is taking up a lot of extra CPU and memory (CPU usage has
doubled and memory is swapping). Is there any documentation on performance
tuning? There seems to be a lot of useful info in the server output but I
don't understand it.

Swapping if it's constant isn't good...  How much memory does this box
have, and what is the heap size of the JVM?  Are there other things
running on this box?

Solr does warming of caches by default to make complex queries that
hit a new snapshot of the index fast.  This takes up CPU in bursts,
but is normally nothing to worry about unless you have other apps
running on the same box that need CPU.  Because of this warming, CPU
usage of a Solr collection isn't directly related to query traffic at
all times.


-Yonik


Reply | Threaded
Open this post in threaded view
|

RE: Performance tuning

Stephanie Belton-2
In reply to this post by Thorsten Scherler-3
Thanks for sending this link, I seem to have missed that on the wiki!

-----Original Message-----
From: Thorsten Scherler [mailto:[hidden email]]
Sent: 11 January 2007 15:06
To: [hidden email]
Subject: Re: Performance tuning

On Thu, 2007-01-11 at 14:57 +0000, Stephanie Belton wrote:

> Hello,
>
>  
>
> Solr is now up and running on our production environment and working great. However it is taking up a lot of extra CPU and memory (CPU usage has doubled and memory is swapping). Is there any documentation on performance tuning? There seems to be a lot of useful info in the server output but I don’t understand it.
>
>  
>
> E.g.        filterCache{lookups=0,hits=0,hitratio=0.00,inserts=537,evictions=0,size=337,cumulative_lookups=4723,cumulative_hits=3708,cumulative_hitratio=0.78,cumulative_inserts=4647,cumulative_evictions=72}
>
>         queryResultCache{lookups=0,hits=0,hitratio=0.00,inserts=256,evictions=0,size=256,cumulative_lookups=3779,cumulative_hits=552,cumulative_hitratio=0.14,cumulative_inserts=3632,cumulative_evictions=0}
>
>         documentCache{lookups=0,hits=0,hitratio=0.00,inserts=0,evictions=0,size=0,cumulative_lookups=66005,cumulative_hits=2460,cumulative_hitratio=0.03,cumulative_inserts=63545,cumulative_evictions=4195}
>
>  
>
> etc. what should I be watching out for?
>

Hi Stephanie,

did you see http://wiki.apache.org/solr/SolrPerformanceFactors?

Further you may consider to balance the load via
http://wiki.apache.org/solr/CollectionDistribution

HTH

salu2

>  
>
> Thanks
>
> Stephanie
>



Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Yonik Seeley-2
In reply to this post by Stephanie Belton-2
On 1/11/07, Stephanie Belton <[hidden email]> wrote:

> This is the output of the free command:
>
> [root@eul0000846 root2]# free -m
>              total       used       free     shared    buffers     cached
> Mem:          2007       1888        119          0         86        814
> -/+ buffers/cache:        986       1020
> Swap:         1992        207       1784
>
> We normally have no swapping at all on this server and since last night
> (when Solr was deployed on the site) it's been going up.

That may be fine... swap in use != swapping.
The OS may be swapping out some processes that haven't been used in a
long time to free up more memory for disk cache (notice 814M cached).
This is a good thing.

> Here is an extract of the top command output sorted by memory usage, does
> each of the processes really take up 566M???

No, older versions of linux show each thread as a separate process.

 CU usage is low because we are
> outside of peak time but during the day it's at 40% when it used to be just
> 20%:

Full-text search is CPU intensive.  An average peak of 40% seems
acceptable.  If the load gets too high, you can scale out by adding
multiple servers behind a load balancer.

-Yonik

> 20:14:16  up 45 days, 21:47,  1 user,  load average: 1.06, 1.14, 1.11
> 167 processes: 166 sleeping, 1 running, 0 zombie, 0 stopped
> CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
>            total    8.8%    0.0%    0.3%   0.1%     0.2%    6.9%   83.2%
>            cpu00    7.9%    0.0%    0.3%   0.7%     0.9%    6.9%   82.8%
>            cpu01    8.5%    0.0%    0.3%   0.0%     0.0%    6.9%   84.0%
>            cpu02    9.9%    0.0%    0.1%   0.0%     0.0%    6.9%   82.8%
>            cpu03    9.0%    0.0%    0.6%   0.0%     0.2%    7.0%   83.2%
> Mem:  2055300k av, 1914588k used,  140712k free,       0k shrd,   89032k
> buff
>                    1326540k actv,  301236k in_d,   30788k in_c
> Swap: 2040244k av,  212948k used, 1827296k free                  843380k
> cached
>
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
> 12201 root      15   0  566M 561M 13276 S     0.0 27.9   0:02   0 java
> 12203 root      15   0  566M 561M 13276 S     0.0 27.9   4:48   2 java
> 12204 root      16   0  566M 561M 13276 S     0.0 27.9   4:45   1 java
> 12205 root      15   0  566M 561M 13276 S     0.0 27.9   4:45   0 java
> 12206 root      15   0  566M 561M 13276 S     0.0 27.9   4:46   2 java
> 12207 root      15   0  566M 561M 13276 S     0.0 27.9   8:35   2 java
> 12208 root      16   0  566M 561M 13276 S     0.0 27.9  15:53   1 java
> 12209 root      16   0  566M 561M 13276 S     0.0 27.9  27:30   1 java
> 12210 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12211 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12212 root      15   0  566M 561M 13276 S     0.0 27.9   0:17   1 java
> 12213 root      15   0  566M 561M 13276 S     0.0 27.9   0:15   2 java
> 12214 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
> 12215 root      15   0  566M 561M 13276 S     0.0 27.9   0:33   2 java
> 12217 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
> 12218 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
> 12219 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12220 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
> 12221 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12222 root      25   0  566M 561M 13276 S     0.0 27.9 297:21   2 java
> 12223 root      15   0  566M 561M 13276 S     0.0 27.9   0:13   3 java
> 12224 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12225 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
> 12226 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
> 12227 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12228 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12229 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12230 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> Etc...
>
> On the server we also have a website running using mod_perl, it's been
> running for 1 year and up until now the CPU usage was peaking at 20% and
> memory around 28% no swapping.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
> Sent: 11 January 2007 15:12
> To: [hidden email]
> Subject: Re: Performance tuning
>
> On 1/11/07, Stephanie Belton <[hidden email]> wrote:
>
> > Solr is now up and running on our production environment and working
> great. However it is taking up a lot of extra CPU and memory (CPU usage has
> doubled and memory is swapping). Is there any documentation on performance
> tuning? There seems to be a lot of useful info in the server output but I
> don't understand it.
>
> Swapping if it's constant isn't good...  How much memory does this box
> have, and what is the heap size of the JVM?  Are there other things
> running on this box?
>
> Solr does warming of caches by default to make complex queries that
> hit a new snapshot of the index fast.  This takes up CPU in bursts,
> but is normally nothing to worry about unless you have other apps
> running on the same box that need CPU.  Because of this warming, CPU
> usage of a Solr collection isn't directly related to query traffic at
> all times.
>
>
> -Yonik
Reply | Threaded
Open this post in threaded view
|

RE: Performance tuning

Stephanie Belton-2
Thanks for that. I am sorry this isn't really Solr-related but how can I
monitor the swapping if I can't rely on the output of the free command?

Do you think I could still achieve any significant improvements by going
through the performance tuning advice on the wiki?

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: 11 January 2007 20:32
To: [hidden email]
Subject: Re: Performance tuning

On 1/11/07, Stephanie Belton <[hidden email]> wrote:

> This is the output of the free command:
>
> [root@eul0000846 root2]# free -m
>              total       used       free     shared    buffers     cached
> Mem:          2007       1888        119          0         86        814
> -/+ buffers/cache:        986       1020
> Swap:         1992        207       1784
>
> We normally have no swapping at all on this server and since last night
> (when Solr was deployed on the site) it's been going up.

That may be fine... swap in use != swapping.
The OS may be swapping out some processes that haven't been used in a
long time to free up more memory for disk cache (notice 814M cached).
This is a good thing.

> Here is an extract of the top command output sorted by memory usage, does
> each of the processes really take up 566M???

No, older versions of linux show each thread as a separate process.

 CU usage is low because we are
> outside of peak time but during the day it's at 40% when it used to be
just
> 20%:

Full-text search is CPU intensive.  An average peak of 40% seems
acceptable.  If the load gets too high, you can scale out by adding
multiple servers behind a load balancer.

-Yonik

> 20:14:16  up 45 days, 21:47,  1 user,  load average: 1.06, 1.14, 1.11
> 167 processes: 166 sleeping, 1 running, 0 zombie, 0 stopped
> CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
>            total    8.8%    0.0%    0.3%   0.1%     0.2%    6.9%   83.2%
>            cpu00    7.9%    0.0%    0.3%   0.7%     0.9%    6.9%   82.8%
>            cpu01    8.5%    0.0%    0.3%   0.0%     0.0%    6.9%   84.0%
>            cpu02    9.9%    0.0%    0.1%   0.0%     0.0%    6.9%   82.8%
>            cpu03    9.0%    0.0%    0.6%   0.0%     0.2%    7.0%   83.2%
> Mem:  2055300k av, 1914588k used,  140712k free,       0k shrd,   89032k
> buff
>                    1326540k actv,  301236k in_d,   30788k in_c
> Swap: 2040244k av,  212948k used, 1827296k free                  843380k
> cached
>
>   PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
> 12201 root      15   0  566M 561M 13276 S     0.0 27.9   0:02   0 java
> 12203 root      15   0  566M 561M 13276 S     0.0 27.9   4:48   2 java
> 12204 root      16   0  566M 561M 13276 S     0.0 27.9   4:45   1 java
> 12205 root      15   0  566M 561M 13276 S     0.0 27.9   4:45   0 java
> 12206 root      15   0  566M 561M 13276 S     0.0 27.9   4:46   2 java
> 12207 root      15   0  566M 561M 13276 S     0.0 27.9   8:35   2 java
> 12208 root      16   0  566M 561M 13276 S     0.0 27.9  15:53   1 java
> 12209 root      16   0  566M 561M 13276 S     0.0 27.9  27:30   1 java
> 12210 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12211 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12212 root      15   0  566M 561M 13276 S     0.0 27.9   0:17   1 java
> 12213 root      15   0  566M 561M 13276 S     0.0 27.9   0:15   2 java
> 12214 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
> 12215 root      15   0  566M 561M 13276 S     0.0 27.9   0:33   2 java
> 12217 root      21   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
> 12218 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
> 12219 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12220 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
> 12221 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12222 root      25   0  566M 561M 13276 S     0.0 27.9 297:21   2 java
> 12223 root      15   0  566M 561M 13276 S     0.0 27.9   0:13   3 java
> 12224 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12225 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   3 java
> 12226 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   2 java
> 12227 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12228 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   0 java
> 12229 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> 12230 root      15   0  566M 561M 13276 S     0.0 27.9   0:00   1 java
> Etc...
>
> On the server we also have a website running using mod_perl, it's been
> running for 1 year and up until now the CPU usage was peaking at 20% and
> memory around 28% no swapping.
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik
Seeley
> Sent: 11 January 2007 15:12
> To: [hidden email]
> Subject: Re: Performance tuning
>
> On 1/11/07, Stephanie Belton <[hidden email]> wrote:
>
> > Solr is now up and running on our production environment and working
> great. However it is taking up a lot of extra CPU and memory (CPU usage
has

> doubled and memory is swapping). Is there any documentation on performance
> tuning? There seems to be a lot of useful info in the server output but I
> don't understand it.
>
> Swapping if it's constant isn't good...  How much memory does this box
> have, and what is the heap size of the JVM?  Are there other things
> running on this box?
>
> Solr does warming of caches by default to make complex queries that
> hit a new snapshot of the index fast.  This takes up CPU in bursts,
> but is normally nothing to worry about unless you have other apps
> running on the same box that need CPU.  Because of this warming, CPU
> usage of a Solr collection isn't directly related to query traffic at
> all times.
>
>
> -Yonik


Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Yonik Seeley-2
On 1/11/07, Stephanie Belton <[hidden email]> wrote:
> Thanks for that. I am sorry this isn't really Solr-related but how can I
> monitor the swapping if I can't rely on the output of the free command?
>
> Do you think I could still achieve any significant improvements by going
> through the performance tuning advice on the wiki?

Unfortunately, I think that's pretty old stuff.

People are normally concerned with:
  - the number of requests per second they can handle with their server
  - the average latency of requests (or median, 99 percentile, etc)

A goal of reducing CPU usage w/o looking at the other factors is
unusual, but if your query rate is very low, or your cache hit rate is
low, you could reduce or eliminate caching or autowarming.

-Yonik
Reply | Threaded
Open this post in threaded view
|

RE: Performance tuning

Stephanie Belton-2
The reason I am keeping a close eye on resource usage is that our traffic is
increasing by around 20% every month (currently over 400,000 page
impressions/day although not all of them are search queries!) and I want to
make sure we tackle any performance issues before it gets too late. I would
rather keep load balancing as a last resort due to cost implications.
 

-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Yonik Seeley
Sent: 11 January 2007 22:02
To: [hidden email]
Subject: Re: Performance tuning

On 1/11/07, Stephanie Belton <[hidden email]> wrote:
> Thanks for that. I am sorry this isn't really Solr-related but how can I
> monitor the swapping if I can't rely on the output of the free command?
>
> Do you think I could still achieve any significant improvements by going
> through the performance tuning advice on the wiki?

Unfortunately, I think that's pretty old stuff.

People are normally concerned with:
  - the number of requests per second they can handle with their server
  - the average latency of requests (or median, 99 percentile, etc)

A goal of reducing CPU usage w/o looking at the other factors is
unusual, but if your query rate is very low, or your cache hit rate is
low, you could reduce or eliminate caching or autowarming.

-Yonik


Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Yonik Seeley-2
On 1/11/07, Stephanie Belton <[hidden email]> wrote:
> The reason I am keeping a close eye on resource usage is that our traffic is
> increasing by around 20% every month (currently over 400,000 page
> impressions/day although not all of them are search queries!) and I want to
> make sure we tackle any performance issues before it gets too late. I would
> rather keep load balancing as a last resort due to cost implications.

Going slightly OT, but if this is business critical, load-balancing
also provides high availability, which can pay for itself in the event
that a server crashes.

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Walter Underwood, Netflix
On 1/11/07 2:33 PM, "Yonik Seeley" <[hidden email]> wrote:

> On 1/11/07, Stephanie Belton <[hidden email]> wrote:
>> The reason I am keeping a close eye on resource usage is that our traffic is
>> increasing by around 20% every month (currently over 400,000 page
>> impressions/day although not all of them are search queries!) and I want to
>> make sure we tackle any performance issues before it gets too late. I would
>> rather keep load balancing as a last resort due to cost implications.
>
> Going slightly OT, but if this is business critical, load-balancing
> also provides high availability, which can pay for itself in the event
> that a server crashes.

Right. For us, load balancing is not a last resort but a fact of life.
The smallest number of parallel servers is three, so that we have
two running when one is down for scheduled maintenance or software
update.

Back to performance, check your cache hit ratios in the admin UI, then
adjust the cache sizes. When your caches are the right size, Solr will
be mostly CPU-bound, but quite fast. If Solr is not CPU-bound under a
maximum load (in testing) it means it is using the disk too much.

wunder
--
Walter Underwood
Search Guru, Netflix


Reply | Threaded
Open this post in threaded view
|

Re: Performance tuning

Mike Klaas
In reply to this post by Stephanie Belton-2
On 1/11/07, Stephanie Belton <[hidden email]> wrote:
> Thanks for that. I am sorry this isn't really Solr-related but how can I
> monitor the swapping if I can't rely on the output of the free command?

$ vmstat -S M 3
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  2      0   2236     34    763    0    0    37    23   77   155  1  0 98  1
 0  1      0   2235     34    763    0    0    71    13  551  2607 16  4 71  9
 1  0      0   2235     34    763    0    0    72   892  742  2194 13  3 67 17

The si/so columns display the real-time swap in/out rates.  vmstat is
also rather useful for all its other columns too.

-Mike
Reply | Threaded
Open this post in threaded view
|

RE: Performance tuning

Stephanie Belton-2
Thank you! Those are set to zero for me too (at the moment!) so I guess it's
good news.

-----Original Message-----
From: Mike Klaas [mailto:[hidden email]]
Sent: 11 January 2007 23:13
To: [hidden email]
Subject: Re: Performance tuning

On 1/11/07, Stephanie Belton <[hidden email]> wrote:
> Thanks for that. I am sorry this isn't really Solr-related but how can I
> monitor the swapping if I can't rely on the output of the free command?

$ vmstat -S M 3
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id
wa
 1  2      0   2236     34    763    0    0    37    23   77   155  1  0 98
1
 0  1      0   2235     34    763    0    0    71    13  551  2607 16  4 71
9
 1  0      0   2235     34    763    0    0    72   892  742  2194 13  3 67
17

The si/so columns display the real-time swap in/out rates.  vmstat is
also rather useful for all its other columns too.

-Mike