Monitor the QTime.

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Monitor the QTime.

Stijn Vanhoorelbeke
Hi,

Is it possible to monitor the QTime of the queries.
I know I could enable logging - but then all of my requests are logged,
making big&nasty logs.

I just want to log the QTime periodically, lets say once every minute.
Is this possible using Solr or can this be set up in tomcat anyway?
Reply | Threaded
Open this post in threaded view
|

Hits when using group=true

Billnbell
It would be good if someone added the hits= on group=true in the log.

We are using this parameter and have build a really cool SOLR log analyzer
(that I am pushing to release to open source).
But it is not as effective if we cannot get group=true to output hits= in
the log - since 90% of our queries are group=true...

There is a ticket in SOLR for this under SOLR-2337. Can someone help me
identify what would be required to get this to work?

Bill

>


Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Gora Mohanty-3
In reply to this post by Stijn Vanhoorelbeke
On Fri, Feb 11, 2011 at 3:40 AM, Stijn Vanhoorelbeke
<[hidden email]> wrote:
> Hi,
>
> Is it possible to monitor the QTime of the queries.
> I know I could enable logging - but then all of my requests are logged,
> making big&nasty logs.
>
> I just want to log the QTime periodically, lets say once every minute.
> Is this possible using Solr or can this be set up in tomcat anyway?

QTime is, of course, specific to the query, but it is returned in the
response XML, so one could run occasional queries to figure it out.
Please see http://wiki.apache.org/solr/SearchHandler

Regards,
Gora
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Ryan McKinley
In reply to this post by Stijn Vanhoorelbeke
You may want to check the stats via JMX.  For example,

http://localhost:8983/solr/core/admin/mbeans?stats=true&key=org.apache.solr.handler.StandardRequestHandler

shows some basic stats info for the handler.

If you are running nagios or similar, they have tools that can log
values from JMX.  this may be helpful:
http://wiki.apache.org/solr/SolrJmx

ryan



On Thu, Feb 10, 2011 at 5:10 PM, Stijn Vanhoorelbeke
<[hidden email]> wrote:
> Hi,
>
> Is it possible to monitor the QTime of the queries.
> I know I could enable logging - but then all of my requests are logged,
> making big&nasty logs.
>
> I just want to log the QTime periodically, lets say once every minute.
> Is this possible using Solr or can this be set up in tomcat anyway?
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Stijn Vanhoorelbeke
In reply to this post by Gora Mohanty-3
> QTime is, of course, specific to the query, but it is returned in the
> response XML, so one could run occasional queries to figure it out.
> Please see http://wiki.apache.org/solr/SearchHandler
>
> Regards,
> Gora
>

Yes, this could be a possibility. But then the Solr cache jumps back into
the picture.
I cannot simply query the system each minute with the same query - that way
the result would be completely satisfied by the internal caches. I could
build a list of heavy queries to do so - but I'd loved to use a more
straight forward method.
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Stijn Vanhoorelbeke
In reply to this post by Ryan McKinley
2011/2/11 Ryan McKinley <[hidden email]>

> You may want to check the stats via JMX.  For example,
>
>
> http://localhost:8983/solr/core/admin/mbeans?stats=true&key=org.apache.solr.handler.StandardRequestHandler
>
> shows some basic stats info for the handler.
> ryan


Can you access this URL from a web browser (tried but doesn't work ) ? Or
must this used in jConsole / custom made java program.

Could you please point me to a good guide to implement this JMX stuff, cause
I'm a newbie for JMX.
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Gora Mohanty-3
In reply to this post by Stijn Vanhoorelbeke
On Sat, Feb 12, 2011 at 3:56 AM, Stijn Vanhoorelbeke
<[hidden email]> wrote:

>> QTime is, of course, specific to the query, but it is returned in the
>> response XML, so one could run occasional queries to figure it out.
>> Please see http://wiki.apache.org/solr/SearchHandler
>>
>> Regards,
>> Gora
>>
>
> Yes, this could be a possibility. But then the Solr cache jumps back into
> the picture.
> I cannot simply query the system each minute with the same query - that way
> the result would be completely satisfied by the internal caches. I could
> build a list of heavy queries to do so - but I'd loved to use a more
> straight forward method.
[...]

That is true. The time for the first query gives you some idea, but that
is just one measurement, and can be off for various reasons. You can
do drastic things like restarting the Java container each time before
querying. However, one should be thinking of what it is that one is trying
to measure, as the benefits of the various Solr/Lucene caches do matter
in operations:
* If it is the average time for real queries in a running system, the stats
  component gives an idea of that, as someone already pointed out. You
  can access this from the admin at http://localhost:8983/solr/admin/stats.jsp
  (hostname/port might be different for you).
* One could pass randomly generated strings to each query so as to take
  out the effect of caching. I have been meaning to hack pg_bench to do
  this. There is a caveat even here, as a query for a non-existent string is
  likely to be faster.

Regards,
Gora
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Gora Mohanty-3
In reply to this post by Stijn Vanhoorelbeke
On Sat, Feb 12, 2011 at 4:54 AM, Stijn Vanhoorelbeke
<[hidden email]> wrote:
[...]
> Can you access this URL from a web browser (tried but doesn't work ) ? Or
> must this used in jConsole / custom made java program.

Please try http://localhost:8983/solr/admin/stats.jsp (change hostname/port as
needed).

> Could you please point me to a good guide to implement this JMX stuff, cause
> I'm a newbie for JMX.

The easiest way to get access to JMX is indeed a Java console, like jconsole.
There are various open-source JMX clients available, but we could find none
that met our needs, and were being actively maintained. We have been
toying with the idea of a JMX client that offers a REST API to Solr MBeans
(or even to any generic MBeans). This would be a more natural interface for
people used to web development.

Regards,
Gora
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Lance Norskog-2
If you're a unix shell scripting wiz, here are a few strategies.

Tail the logfile and filter for the string 'QTime'. The number is the
very last string in the line. So, strip the text between the timestamp
and the number- sort by the timestamp first and the number second. Now
 grab the first qtime for each timestamp. I don't know a command for
this. This gives you the longest query time for each second.

As a separate trick: tail the logfile and filter for QTime. Then,
strip out all text after the time. Now you have a stream of lines with
a timestamp. Run this through 'uniq -c' and voila! you get the queries
per second for each timestamp.

On Sat, Feb 12, 2011 at 1:51 AM, Gora Mohanty <[hidden email]> wrote:

> On Sat, Feb 12, 2011 at 4:54 AM, Stijn Vanhoorelbeke
> <[hidden email]> wrote:
> [...]
>> Can you access this URL from a web browser (tried but doesn't work ) ? Or
>> must this used in jConsole / custom made java program.
>
> Please try http://localhost:8983/solr/admin/stats.jsp (change hostname/port as
> needed).
>
>> Could you please point me to a good guide to implement this JMX stuff, cause
>> I'm a newbie for JMX.
>
> The easiest way to get access to JMX is indeed a Java console, like jconsole.
> There are various open-source JMX clients available, but we could find none
> that met our needs, and were being actively maintained. We have been
> toying with the idea of a JMX client that offers a REST API to Solr MBeans
> (or even to any generic MBeans). This would be a more natural interface for
> people used to web development.
>
> Regards,
> Gora
>



--
Lance Norskog
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Otis Gospodnetic-2
In reply to this post by Gora Mohanty-3
I've recently used a jmx console tool recently, which let's you access and
browse jmx from a simple shell that you start from command line.  Took me 5
minutes to get it and get info from jmx.

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



----- Original Message ----

> From: Gora Mohanty <[hidden email]>
> To: [hidden email]
> Sent: Sat, February 12, 2011 4:51:48 AM
> Subject: Re: Monitor the QTime.
>
> On Sat, Feb 12, 2011 at 4:54 AM, Stijn Vanhoorelbeke
> <[hidden email]>  wrote:
> [...]
> > Can you access this URL from a web browser (tried but  doesn't work ) ? Or
> > must this used in jConsole / custom made java  program.
>
> Please try http://localhost:8983/solr/admin/stats.jsp (change  hostname/port
as
> needed).
>
> > Could you please point me to a good  guide to implement this JMX stuff,
cause

> > I'm a newbie for  JMX.
>
> The easiest way to get access to JMX is indeed a Java console, like  jconsole.
> There are various open-source JMX clients available, but we could  find none
> that met our needs, and were being actively maintained. We have  been
> toying with the idea of a JMX client that offers a REST API to Solr  MBeans
> (or even to any generic MBeans). This would be a more natural  interface for
> people used to web development.
>
> Regards,
> Gora
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Gora Mohanty-3
On Mon, Feb 14, 2011 at 8:46 AM, Otis Gospodnetic
<[hidden email]> wrote:
> I've recently used a jmx console tool recently, which let's you access and
> browse jmx from a simple shell that you start from command line.  Took me 5
> minutes to get it and get info from jmx.
[...]

Could you share which tool this is? There are Nagios plugins
that do that, and they are indeed useful. I would still like to
have a REST API on top of JMX. Hopefully, we will be able
to do that soon.

Regards,
Gora
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Gora Mohanty-3
In reply to this post by Lance Norskog-2
On Sun, Feb 13, 2011 at 7:54 AM, Lance Norskog <[hidden email]> wrote:
> If you're a unix shell scripting wiz, here are a few strategies.
>
> Tail the logfile and filter for the string 'QTime'. The number is the
> very last string in the line. So, strip the text between the timestamp
> and the number- sort by the timestamp first and the number second. Now
>  grab the first qtime for each timestamp. I don't know a command for
> this. This gives you the longest query time for each second.
[...]

Thanks for the idea. As this was useful enough for us, have gone ahead
and implemented it.

* Attached is a bash script that will print
    timestamp query-string qtime
  for each query in a list of Solr log files specified as command-line
  arguments.
* Use as "./qtime_solr.sh logfile1 logfile2...". Without any arguments,
  the script uses /var/log/tomcat6/catalina.out as the log file.
* Have only minimally tested it, but would be glad to fix any bugs
  that people encounter.
* Have *not* implemented the latter part of the suggestion, i.e.,
  sorting by timestamp, and taking only the first query. IMHO, the
  needs here might be different, and it is best to have a second
  script doing this, that uses the output from this one.
* Not sure if attachments to this list are stripped, so the script has
  also been uploaded to http://pastebin.com/YLqBHp19

Regards,
Gora

qtime_solr.sh (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Monitor the QTime.

Otis Gospodnetic-2
In reply to this post by Gora Mohanty-3
Hi,

I think the tool was called jmxterm or termjmx.  As for REST over JMX - I think
I've seen that on Google code.  If you are interested in this sort of stuff,
check my bookmarks on pinboard.in, it's all there and nicely tagged.

Otis
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch
Lucene ecosystem search :: http://search-lucene.com/



----- Original Message ----

> From: Gora Mohanty <[hidden email]>
> To: [hidden email]
> Sent: Sun, February 13, 2011 11:14:04 PM
> Subject: Re: Monitor the QTime.
>
> On Mon, Feb 14, 2011 at 8:46 AM, Otis Gospodnetic
> <[hidden email]>  wrote:
> > I've recently used a jmx console tool recently, which let's you  access and
> > browse jmx from a simple shell that you start from command  line.  Took me 5
> > minutes to get it and get info from  jmx.
> [...]
>
> Could you share which tool this is? There are Nagios  plugins
> that do that, and they are indeed useful. I would still like  to
> have a REST API on top of JMX. Hopefully, we will be able
> to do that  soon.
>
> Regards,
> Gora
>