Re: Requests per second/minute monitor?

classic Classic list List threaded Threaded
25 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Otis Gospodnetic-2
Would creating a new QueryRateFilter servlet filter be a good place to put this?  This way it could stay out of the Solr core and coult be turned on/off via web.xml.

Otis

----- Original Message ----
From: Chris Hostetter <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Thursday, April 19, 2007 5:37:56 PM
Subject: Re: Requests per second/minute monitor?


: Is there a good spot to track request rate in Solr? Has anyone
: built a monitor?

I would think it would make more sense to track this in your application
server then to add it to Solr itself.




-Hoss




Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Yonik Seeley-2
On 4/27/07, Otis Gospodnetic <[hidden email]> wrote:
> Would creating a new QueryRateFilter servlet filter be a good place to put this?  This way it could stay out of the Solr core and coult be turned on/off via web.xml.

There's already gotta be some nice external tools that parse log files
and produce pretty graphs, no?

-Yonik
Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Walter Underwood, Netflix
This is for monitoring -- what happened in the last 30 seconds.
Log file analysis doesn't really do that.

I think the XML output in admin/stats.jsp may be enough for us.
That gives the cumulative requests on each handler. Those are
counted in StandardRequestHandler DisMaxRequestHandler and
are available through the MBean interface.

wunder

On 4/27/07 1:15 PM, "Yonik Seeley" <[hidden email]> wrote:

> On 4/27/07, Otis Gospodnetic <[hidden email]> wrote:
>> Would creating a new QueryRateFilter servlet filter be a good place to put
>> this?  This way it could stay out of the Solr core and coult be turned on/off
>> via web.xml.
>
> There's already gotta be some nice external tools that parse log files
> and produce pretty graphs, no?
>
> -Yonik

Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Ryan McKinley
Walter Underwood wrote:
> This is for monitoring -- what happened in the last 30 seconds.
> Log file analysis doesn't really do that.
>
> I think the XML output in admin/stats.jsp may be enough for us.
> That gives the cumulative requests on each handler. Those are
> counted in StandardRequestHandler DisMaxRequestHandler and
> are available through the MBean interface.
>

If you are running a solr build since yesterday, take a look at the
PluginInfoHandler

  http://localhost:8983/solr/admin/plugins?stats=true

This gives you standard response format access to the same info.

I'll write up some docs for the wiki shortly

Perhaps it should be modified to let you specify a single handler.
Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Otis Gospodnetic-2
In reply to this post by Otis Gospodnetic-2
I think the real-time-ness of this is the key.  What's the current QPS?  How many in-flight queries do we have?  What is the average or mean response time?  What's the response time for the 90% percentile? etc.  Anyhow, not my current itch, just trying to point out what Wunder is after.

Otis


----- Original Message ----
From: Yonik Seeley <[hidden email]>
To: [hidden email]
Sent: Friday, April 27, 2007 4:15:33 PM
Subject: Re: Requests per second/minute monitor?

On 4/27/07, Otis Gospodnetic <[hidden email]> wrote:
> Would creating a new QueryRateFilter servlet filter be a good place to put this?  This way it could stay out of the Solr core and coult be turned on/off via web.xml.

There's already gotta be some nice external tools that parse log files
and produce pretty graphs, no?

-Yonik



Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Ian Holsman (Lists)
In reply to this post by Ryan McKinley

Walter Underwood wrote:
> This is for monitoring -- what happened in the last 30 seconds.
> Log file analysis doesn't really do that.
>

I would respectfully disagree.
Log file analysis of each request can give you that, and a whole lot more.

you could either grab the stats via a regular cron job, or create a separate filter to parse them real time.
It would then let you grab more sophisticated stats if you choose to.

What I would like to know is (and excuse the newbieness of the question) how to enable solr to log a file with the following data.


- time spent (ms) in the request.
- IP# of the incoming request
- what the request was (and what handler executed it)
- a status code to signal if the request failed for some reasons
- number of rows fetched
and
- the number of rows actually returned

is this possible? (I'm using tomcat if that changes the answer).

regards
Ian
Reply | Threaded
Open this post in threaded view
|

ktnxbye

Maarten.De.Vilder
hey

we (me and my fellow student) have been on an intership the last months

we had to make a search function for an online shop and we used Solr

since the shop is not openly available for everybody, we made a demo video
for everyone to enjoy

we'ld like to thank al the people at Solr/Lucene and everyone who helped
us out over here

enjoy the show : http://deepthought.be/demovid.wmv 

ktnxbye!
Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Walter Underwood, Netflix
In reply to this post by Ian Holsman (Lists)
Yes, that is possible, but we also monitor Apache, Tomcat, the JVM, and
OS through JMX and other live monitoring interfaces. Why invent a real-time
HTTP log analysis system when I can fetch /search/stats.jsp at any time?

By "number of rows fetched", do you mean "number of documents matched"?

The log you describe is pretty useful. Ultraseek has something similar
and that is the log most often used by admins. I'd recommend also
logging the start and rows part of the request so you can distinguish
between new queries and second page requests. If possible, make the
timestamp the same as the HTTP access log so you can correlate the
entries.

wunder

On 5/9/07 9:43 PM, "Ian Holsman" <[hidden email]> wrote:

>
> Walter Underwood wrote:
>> This is for monitoring -- what happened in the last 30 seconds.
>> Log file analysis doesn't really do that.
>
> I would respectfully disagree.
> Log file analysis of each request can give you that, and a whole lot more.
>
> you could either grab the stats via a regular cron job, or create a separate
> filter to parse them real time.
> It would then let you grab more sophisticated stats if you choose to.
>
> What I would like to know is (and excuse the newbieness of the question) how
> to enable solr to log a file with the following data.
>
> - time spent (ms) in the request.
> - IP# of the incoming request
> - what the request was (and what handler executed it)
> - a status code to signal if the request failed for some reasons
> - number of rows fetched
> and
> - the number of rows actually returned
>
> is this possible? (I'm using tomcat if that changes the answer).
>
> regards
> Ian

Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Ian Holsman (Lists)
Walter Underwood wrote:
> Yes, that is possible, but we also monitor Apache, Tomcat, the JVM, and
> OS through JMX and other live monitoring interfaces. Why invent a real-time
> HTTP log analysis system when I can fetch /search/stats.jsp at any time?
>  
 "there are lies, damnd lies, and statistics"

The stats page, while useful, can be dangerous if you don't fully
understand what you are looking at and it's limitations.

3 examples spring to mind.

1. a development group fetching 20k of records at a time every hour or
so off a production server.
This would represent itself on by the average rows returned going to 10.
So at the casual glance,
it looked like every request was returning 10, and the dev request was
hidden.

2. the average isn't broken down enough. certain queries are more
complex than others, and usually executed
less often than the simple ones.  These skew the numbers, and hide the
bad performers

3. variation is the key to performance, not averages IMHO. I would
prefer responses delivered slower, but with no
variations (ie it will always take 2ms), then generally faster queries
which bite every once and a while. Averages
dont show you this.


ok.. thats my rant about averages gone. now of course you could always
implement variation and percentile performance
figures inside of SolR and show them on the stats page, but this can get
complex quickly.
> By "number of rows fetched", do you mean "number of documents matched"?
>  
yep, I just grabbed the log format of a similar search appliance we use,
It would be better to add solr/lucene specific
things to it. I'm not tied to it.
> The log you describe is pretty useful. Ultraseek has something similar
> and that is the log most often used by admins. I'd recommend also
> logging the start and rows part of the request so you can distinguish
> between new queries and second page requests. If possible, make the
> timestamp the same as the HTTP access log so you can correlate the
> entries.
>  
yep, that's a good idea as well.

I tinkered a bit more with the idea last night and came up with
https://issues.apache.org/jira/browse/SOLR-232
which allows a tomcat container access to somewhere we can put the data
we need from the request.

feedback is welcome, as I defiantly don't have all the answers, and am a
newbie when it comes to solr ;-)

--Ian

> wunder
>
> On 5/9/07 9:43 PM, "Ian Holsman" <[hidden email]> wrote:
>  
>> Walter Underwood wrote:
>>    
>>> This is for monitoring -- what happened in the last 30 seconds.
>>> Log file analysis doesn't really do that.
>>>      
>> I would respectfully disagree.
>> Log file analysis of each request can give you that, and a whole lot more.
>>
>> you could either grab the stats via a regular cron job, or create a separate
>> filter to parse them real time.
>> It would then let you grab more sophisticated stats if you choose to.
>>
>> What I would like to know is (and excuse the newbieness of the question) how
>> to enable solr to log a file with the following data.
>>
>> - time spent (ms) in the request.
>> - IP# of the incoming request
>> - what the request was (and what handler executed it)
>> - a status code to signal if the request failed for some reasons
>> - number of rows fetched
>> and
>> - the number of rows actually returned
>>
>> is this possible? (I'm using tomcat if that changes the answer).
>>
>> regards
>> Ian
>>    
>
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Yonik Seeley-2
In reply to this post by Ian Holsman (Lists)
On 5/10/07, Ian Holsman <[hidden email]> wrote:
> What I would like to know is (and excuse the newbieness of the question) how
> to enable solr to log a file with the following data.
>
>
> - time spent (ms) in the request.

currently logged

> - IP# of the incoming request

normally in the container access log?

> - what the request was (and what handler executed it)

currently logged

> - a status code to signal if the request failed for some reasons

currently logged

> - number of rows fetched

The number of documents that matched?  That's a higher level concept
rather specific to a request handler.   That info is returned in most
responses though.

> and
> - the number of rows actually returned

That's also in the response, but would be largely meaningless in general.
One could also determine this number from the input parameters and the
number of docs that matched.

A better number might be size of the response (which is normally in
the container access log).
fields could be very small, or very large, and faceting, highlighting,
or other data could dwarf the size/speed due to the main response
documents.

-Yonik
Reply | Threaded
Open this post in threaded view
|

RE: Requests per second/minute monitor?

Will Johnson
I've needed similar logged information recently and I looked at the code
and had a few questions:

Why does SolrCore.setResponseHeaderValues(...) set the QTime (and other
response header options) instead of having it as a function of
RequestHandlerBase?  If things were tracked in the RequestHandlers you
could add timing information there: avg query time, etc.  I know some
people have argued that you can do that with logs but being able to pull
that info live via JMX/stats.jsp would make monitoring much cleaner in
environments with multiple machines on different networks.  If things
are tracked in the handlers then people can add more statistics easily
to both query response headers and overall via custom handlers.

I'm happy to make the changes and supply a patch to move the logic as
well as adding a few simple metrics unless enough people on this thread
really feel that it's always better to do it with log files and
postmortem math.

- will    
Reply | Threaded
Open this post in threaded view
|

RE: Requests per second/minute monitor?

Chris Hostetter-3

: Why does SolrCore.setResponseHeaderValues(...) set the QTime (and other
: response header options) instead of having it as a function of

the main reason was to ensure that that data would *always* be there no
matter who wrote a request handler (or wether or not they subclassed the
RequestHandlerBase ... it's a backwards compatibility issue, we want to
ensure that data even if you have a custom request handler you've been
using since Solr 1.1 (or earlier)

: RequestHandlerBase?  If things were tracked in the RequestHandlers you
: could add timing information there: avg query time, etc.  I know some

you still can track those things, and return that data or log that data as
well ... although if you really wnated to know how long the *whole*
request took you would need to do it in teh ResponseWriter (or in the
core, after the response has been written)

: I'm happy to make the changes and supply a patch to move the logic as
: well as adding a few simple metrics unless enough people on this thread
: really feel that it's always better to do it with log files and
: postmortem math.

moving the logic would be bad .. adding new helper utilities to the
RequestHandler Base for handlers that want to do more sounds fine to me.

see also http://issues.apache.org/jira/browse/SOLR-176 which already adds
a lot of timing info to requests.


-Hoss

Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Mike Klaas

On 15-May-07, at 1:41 PM, Chris Hostetter wrote:

> : I'm happy to make the changes and supply a patch to move the  
> logic as
> : well as adding a few simple metrics unless enough people on this  
> thread
> : really feel that it's always better to do it with log files and
> : postmortem math.
>
> moving the logic would be bad .. adding new helper utilities to the
> RequestHandler Base for handlers that want to do more sounds fine  
> to me.
>
> see also http://issues.apache.org/jira/browse/SOLR-176 which  
> already adds
> a lot of timing info to requests.

Yeah, I held off on that as it seemed that timing/statistics might be  
handleable on a larger scale.  OTOH, it does give an easy way to  
requesthandlers to insert detailed timing data in a logical place in  
the output.

-Mike
Reply | Threaded
Open this post in threaded view
|

RE: Requests per second/minute monitor?

Will Johnson
Adding entries to RequestHandlerBase.getStatistics() sounds like it
might be a reasonable compromise; backwards compatibility is kept in
place but everything from now on gets the added advantages of more
tracking.  So far I've added (because I need)

avgTimePerRequest
avgRequestsPerSecond

I agree that you can pull this kind of information from logs but jmx
accessibility + the built in stats page make life a good bit simpler
especially in distributed environments.

Are there any others that people think would be nice to have or should I
stop all together?

- will

-----Original Message-----
From: Mike Klaas [mailto:[hidden email]]
Sent: Tuesday, May 15, 2007 5:04 PM
To: [hidden email]
Subject: Re: Requests per second/minute monitor?


On 15-May-07, at 1:41 PM, Chris Hostetter wrote:

> : I'm happy to make the changes and supply a patch to move the  
> logic as
> : well as adding a few simple metrics unless enough people on this  
> thread
> : really feel that it's always better to do it with log files and
> : postmortem math.
>
> moving the logic would be bad .. adding new helper utilities to the
> RequestHandler Base for handlers that want to do more sounds fine  
> to me.
>
> see also http://issues.apache.org/jira/browse/SOLR-176 which  
> already adds
> a lot of timing info to requests.

Yeah, I held off on that as it seemed that timing/statistics might be  
handleable on a larger scale.  OTOH, it does give an easy way to  
requesthandlers to insert detailed timing data in a logical place in  
the output.

-Mike
Reply | Threaded
Open this post in threaded view
|

PriceJunkie.com using solr!

maustin

I just wanted to say thanks to everyone for the creation of solr.  I've been
using it for a while now and I have recently brought one of my side projects
online.  I have several other projects that will be using solr for it's
search and facets.

Please check out www.pricejunkie.com and let us know what you think.. You
can give feedback and/or sign up on the mailing list for future updates.
The site is very basic right now and many new and useful features plus
merchants and product categories will be coming soon!  I thought it would be
a good idea to at least have a few people use it to get some feedback early
and often.

Some of the nice things behind the scenes that we did with solr:
- created custom request handlers that have category to facet to attribute
caching built in
- category to facet management
        - ability to manage facet groups (attributes within a set facet) and assign
them to categories
        - ability to create any category structure and share facet groups

- facet inheritance for any category (a facet group can be defined on a
parent category and pushed down to all children)
- ability to create sub-categories as facets instead of normal sub
categories
- simple xml configuration for the final outputted category configuration
file


I'm sure there are more cool things but that is all for now.  Join the
mailing list to see more improvements in the future.

Also.. how do I get added to the Using Solr wiki page?


Thanks,
Mike Austin

Reply | Threaded
Open this post in threaded view
|

Re: PriceJunkie.com using solr!

Yonik Seeley-2
Congrats, very nice job!
It's fast too.

-Yonik

On 5/16/07, Mike Austin <[hidden email]> wrote:

> I just wanted to say thanks to everyone for the creation of solr.  I've been
> using it for a while now and I have recently brought one of my side projects
> online.  I have several other projects that will be using solr for it's
> search and facets.
>
> Please check out www.pricejunkie.com and let us know what you think.. You
> can give feedback and/or sign up on the mailing list for future updates.
> The site is very basic right now and many new and useful features plus
> merchants and product categories will be coming soon!  I thought it would be
> a good idea to at least have a few people use it to get some feedback early
> and often.
>
> Some of the nice things behind the scenes that we did with solr:
> - created custom request handlers that have category to facet to attribute
> caching built in
> - category to facet management
>         - ability to manage facet groups (attributes within a set facet) and assign
> them to categories
>         - ability to create any category structure and share facet groups
>
> - facet inheritance for any category (a facet group can be defined on a
> parent category and pushed down to all children)
> - ability to create sub-categories as facets instead of normal sub
> categories
> - simple xml configuration for the final outputted category configuration
> file
>
>
> I'm sure there are more cool things but that is all for now.  Join the
> mailing list to see more improvements in the future.
>
> Also.. how do I get added to the Using Solr wiki page?
>
>
> Thanks,
> Mike Austin
Reply | Threaded
Open this post in threaded view
|

Re: PriceJunkie.com using solr!

James liu-2
how many solr instance?


2007/5/17, Yonik Seeley <[hidden email]>:

>
> Congrats, very nice job!
> It's fast too.
>
> -Yonik
>
> On 5/16/07, Mike Austin <[hidden email]> wrote:
> > I just wanted to say thanks to everyone for the creation of solr.  I've
> been
> > using it for a while now and I have recently brought one of my side
> projects
> > online.  I have several other projects that will be using solr for it's
> > search and facets.
> >
> > Please check out www.pricejunkie.com and let us know what you think..
> You
> > can give feedback and/or sign up on the mailing list for future updates.
> > The site is very basic right now and many new and useful features plus
> > merchants and product categories will be coming soon!  I thought it
> would be
> > a good idea to at least have a few people use it to get some feedback
> early
> > and often.
> >
> > Some of the nice things behind the scenes that we did with solr:
> > - created custom request handlers that have category to facet to
> attribute
> > caching built in
> > - category to facet management
> >         - ability to manage facet groups (attributes within a set facet)
> and assign
> > them to categories
> >         - ability to create any category structure and share facet
> groups
> >
> > - facet inheritance for any category (a facet group can be defined on a
> > parent category and pushed down to all children)
> > - ability to create sub-categories as facets instead of normal sub
> > categories
> > - simple xml configuration for the final outputted category
> configuration
> > file
> >
> >
> > I'm sure there are more cool things but that is all for now.  Join the
> > mailing list to see more improvements in the future.
> >
> > Also.. how do I get added to the Using Solr wiki page?
> >
> >
> > Thanks,
> > Mike Austin
>



--
regards
jl
Reply | Threaded
Open this post in threaded view
|

Re: PriceJunkie.com using solr!

martin.grotzke
In reply to this post by maustin
Very nice and really fast, congrats!

Are you willing to provide the mentioned features to solr users?
I think espacially the category to facet management (facet groups)
is really useful...
It would be very nice to have this problem solved once... :)

Cheers,
Martin


On Wed, 2007-05-16 at 16:28 -0500, Mike Austin wrote:

> I just wanted to say thanks to everyone for the creation of solr.  I've been
> using it for a while now and I have recently brought one of my side projects
> online.  I have several other projects that will be using solr for it's
> search and facets.
>
> Please check out www.pricejunkie.com and let us know what you think.. You
> can give feedback and/or sign up on the mailing list for future updates.
> The site is very basic right now and many new and useful features plus
> merchants and product categories will be coming soon!  I thought it would be
> a good idea to at least have a few people use it to get some feedback early
> and often.
>
> Some of the nice things behind the scenes that we did with solr:
> - created custom request handlers that have category to facet to attribute
> caching built in
> - category to facet management
> - ability to manage facet groups (attributes within a set facet) and assign
> them to categories
> - ability to create any category structure and share facet groups
>
> - facet inheritance for any category (a facet group can be defined on a
> parent category and pushed down to all children)
> - ability to create sub-categories as facets instead of normal sub
> categories
> - simple xml configuration for the final outputted category configuration
> file
>
>
> I'm sure there are more cool things but that is all for now.  Join the
> mailing list to see more improvements in the future.
>
> Also.. how do I get added to the Using Solr wiki page?
>
>
> Thanks,
> Mike Austin
>
--
Martin Grotzke
http://www.javakaffee.de/blog/

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Requests per second/minute monitor?

Otis Gospodnetic-2
In reply to this post by Otis Gospodnetic-2
I haven't looked nor tried your patch (just saw it come into JIRA) yet, but I would welcome having stats in one place and real-time instead of (really: in addition to) analyzing logs.

Otis


----- Original Message ----
From: Will Johnson <[hidden email]>
To: [hidden email]
Sent: Wednesday, May 16, 2007 2:02:12 PM
Subject: RE: Requests per second/minute monitor?

Adding entries to RequestHandlerBase.getStatistics() sounds like it
might be a reasonable compromise; backwards compatibility is kept in
place but everything from now on gets the added advantages of more
tracking.  So far I've added (because I need)

avgTimePerRequest
avgRequestsPerSecond

I agree that you can pull this kind of information from logs but jmx
accessibility + the built in stats page make life a good bit simpler
especially in distributed environments.

Are there any others that people think would be nice to have or should I
stop all together?

- will

-----Original Message-----
From: Mike Klaas [mailto:[hidden email]]
Sent: Tuesday, May 15, 2007 5:04 PM
To: [hidden email]
Subject: Re: Requests per second/minute monitor?


On 15-May-07, at 1:41 PM, Chris Hostetter wrote:

> : I'm happy to make the changes and supply a patch to move the  
> logic as
> : well as adding a few simple metrics unless enough people on this  
> thread
> : really feel that it's always better to do it with log files and
> : postmortem math.
>
> moving the logic would be bad .. adding new helper utilities to the
> RequestHandler Base for handlers that want to do more sounds fine  
> to me.
>
> see also http://issues.apache.org/jira/browse/SOLR-176 which  
> already adds
> a lot of timing info to requests.

Yeah, I held off on that as it seemed that timing/statistics might be  
handleable on a larger scale.  OTOH, it does give an easy way to  
requesthandlers to insert detailed timing data in a logical place in  
the output.

-Mike



Reply | Threaded
Open this post in threaded view
|

Re: PriceJunkie.com using solr!

Tim Archambault
In reply to this post by maustin
I did a search and noticed pages were executed through aspx. Are you using
.net to parse the xml results from SOLR? Nice site, just trying to figure
out where SOLR fits into this.

On 5/16/07, Mike Austin <[hidden email]> wrote:

>
>
> I just wanted to say thanks to everyone for the creation of solr.  I've
> been
> using it for a while now and I have recently brought one of my side
> projects
> online.  I have several other projects that will be using solr for it's
> search and facets.
>
> Please check out www.pricejunkie.com and let us know what you think.. You
> can give feedback and/or sign up on the mailing list for future updates.
> The site is very basic right now and many new and useful features plus
> merchants and product categories will be coming soon!  I thought it would
> be
> a good idea to at least have a few people use it to get some feedback
> early
> and often.
>
> Some of the nice things behind the scenes that we did with solr:
> - created custom request handlers that have category to facet to attribute
> caching built in
> - category to facet management
>         - ability to manage facet groups (attributes within a set facet)
> and assign
> them to categories
>         - ability to create any category structure and share facet groups
>
> - facet inheritance for any category (a facet group can be defined on a
> parent category and pushed down to all children)
> - ability to create sub-categories as facets instead of normal sub
> categories
> - simple xml configuration for the final outputted category configuration
> file
>
>
> I'm sure there are more cool things but that is all for now.  Join the
> mailing list to see more improvements in the future.
>
> Also.. how do I get added to the Using Solr wiki page?
>
>
> Thanks,
> Mike Austin
>
>
12