Need help with Slow Query Logging

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Need help with Slow Query Logging

Atita Arora
Hi ,

I have a situation here where I am required to log the slow queries into a
seperate log file which then can be used for optimization purposes.
For now this log is aggregated into the mainstream log marking
[slow:......].
I looked into the code and the configuration and I am really clueless as to
how do I go about seperating the slow query logs as it needs another file
appender
to be created other than the one already present in the log4j.
If I create another appender I can do so by degregating through log levels
, so that moves all the WARN logs to another file (which is not what I am
looking for).
Also from the code prespective , I feel how about if I introduce another
config setting along with the slowQueryThresholdMillis value , something
like

slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);


where slowQueryLogFile and if present it logs into this file otherwise it
works on the already present along with

slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis", -1);


or should I tweak log4j ?
I am not sure if anyone has done that before or have any pointers to guide
me on this.
Please help.

Thanks in advance,
Atita
Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Emir Arnautović
Hi Atita,
You should definetely go with log4j configuration as anything else would be redoing what log4j can do. You already have slowQueryThresholdMillies to make slow queries log with WARN and you can configure log4j to put such logs (class + level) to a separate file.
This seems like frequent question and not sure why putting logs to separate file is not a default configuration - maybe it would make things bit more complicated with logs view in admin console…
If get stuck, let me know (+ Solr version) and I’ll play a bit and send you configs.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
>
> Hi ,
>
> I have a situation here where I am required to log the slow queries into a
> seperate log file which then can be used for optimization purposes.
> For now this log is aggregated into the mainstream log marking
> [slow:......].
> I looked into the code and the configuration and I am really clueless as to
> how do I go about seperating the slow query logs as it needs another file
> appender
> to be created other than the one already present in the log4j.
> If I create another appender I can do so by degregating through log levels
> , so that moves all the WARN logs to another file (which is not what I am
> looking for).
> Also from the code prespective , I feel how about if I introduce another
> config setting along with the slowQueryThresholdMillis value , something
> like
>
> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
>
>
> where slowQueryLogFile and if present it logs into this file otherwise it
> works on the already present along with
>
> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis", -1);
>
>
> or should I tweak log4j ?
> I am not sure if anyone has done that before or have any pointers to guide
> me on this.
> Please help.
>
> Thanks in advance,
> Atita

Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Atita Arora
Hi Emir,

So I made few changes to the log4j config , I am able to redirect these
logs to another file as well.
But as these are the WARN logs so I doubt any logs enabled at WARN level
are going to be redirected here in this new log file.
So precisely , I am using Solr 6.1 (in cloud mode) & I have made few more
changes to the logging levels and components.
Please find my log4j at : *https://pastebin.com/uTLAiBE5
<https://pastebin.com/uTLAiBE5>*

Any help on this will surely be appreciated.

Thanks again.

Atita


On Tue, Oct 10, 2017 at 1:39 PM, Emir Arnautović <
[hidden email]> wrote:

> Hi Atita,
> You should definetely go with log4j configuration as anything else would
> be redoing what log4j can do. You already have slowQueryThresholdMillies to
> make slow queries log with WARN and you can configure log4j to put such
> logs (class + level) to a separate file.
> This seems like frequent question and not sure why putting logs to
> separate file is not a default configuration - maybe it would make things
> bit more complicated with logs view in admin console…
> If get stuck, let me know (+ Solr version) and I’ll play a bit and send
> you configs.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
> >
> > Hi ,
> >
> > I have a situation here where I am required to log the slow queries into
> a
> > seperate log file which then can be used for optimization purposes.
> > For now this log is aggregated into the mainstream log marking
> > [slow:......].
> > I looked into the code and the configuration and I am really clueless as
> to
> > how do I go about seperating the slow query logs as it needs another file
> > appender
> > to be created other than the one already present in the log4j.
> > If I create another appender I can do so by degregating through log
> levels
> > , so that moves all the WARN logs to another file (which is not what I am
> > looking for).
> > Also from the code prespective , I feel how about if I introduce another
> > config setting along with the slowQueryThresholdMillis value , something
> > like
> >
> > slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
> >
> >
> > where slowQueryLogFile and if present it logs into this file otherwise it
> > works on the already present along with
> >
> > slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis", -1);
> >
> >
> > or should I tweak log4j ?
> > I am not sure if anyone has done that before or have any pointers to
> guide
> > me on this.
> > Please help.
> >
> > Thanks in advance,
> > Atita
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Emir Arnautović
Hi Atita,
I did not try it, but I think that following could work:


#logging queries
log4j.logger.org.apache.solr.handler.component.QueryComponent=WARN,slow
 
log4j.appender.slow=org.apache.log4j.RollingFileAppender
log4j.appender.slow.File=${solr.log}/slow.log
log4j.appender.slow.layout=org.apache.log4j.EnhancedPatternLayout
log4j.appender.slow.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n

If you want to log all queries, you can change level for query component to INFO.

HTH,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 10 Oct 2017, at 13:35, Atita Arora <[hidden email]> wrote:
>
> Hi Emir,
>
> So I made few changes to the log4j config , I am able to redirect these
> logs to another file as well.
> But as these are the WARN logs so I doubt any logs enabled at WARN level
> are going to be redirected here in this new log file.
> So precisely , I am using Solr 6.1 (in cloud mode) & I have made few more
> changes to the logging levels and components.
> Please find my log4j at : *https://pastebin.com/uTLAiBE5
> <https://pastebin.com/uTLAiBE5>*
>
> Any help on this will surely be appreciated.
>
> Thanks again.
>
> Atita
>
>
> On Tue, Oct 10, 2017 at 1:39 PM, Emir Arnautović <
> [hidden email]> wrote:
>
>> Hi Atita,
>> You should definetely go with log4j configuration as anything else would
>> be redoing what log4j can do. You already have slowQueryThresholdMillies to
>> make slow queries log with WARN and you can configure log4j to put such
>> logs (class + level) to a separate file.
>> This seems like frequent question and not sure why putting logs to
>> separate file is not a default configuration - maybe it would make things
>> bit more complicated with logs view in admin console…
>> If get stuck, let me know (+ Solr version) and I’ll play a bit and send
>> you configs.
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>>> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
>>>
>>> Hi ,
>>>
>>> I have a situation here where I am required to log the slow queries into
>> a
>>> seperate log file which then can be used for optimization purposes.
>>> For now this log is aggregated into the mainstream log marking
>>> [slow:......].
>>> I looked into the code and the configuration and I am really clueless as
>> to
>>> how do I go about seperating the slow query logs as it needs another file
>>> appender
>>> to be created other than the one already present in the log4j.
>>> If I create another appender I can do so by degregating through log
>> levels
>>> , so that moves all the WARN logs to another file (which is not what I am
>>> looking for).
>>> Also from the code prespective , I feel how about if I introduce another
>>> config setting along with the slowQueryThresholdMillis value , something
>>> like
>>>
>>> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
>>>
>>>
>>> where slowQueryLogFile and if present it logs into this file otherwise it
>>> works on the already present along with
>>>
>>> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis", -1);
>>>
>>>
>>> or should I tweak log4j ?
>>> I am not sure if anyone has done that before or have any pointers to
>> guide
>>> me on this.
>>> Please help.
>>>
>>> Thanks in advance,
>>> Atita
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Atita Arora
Sure thanks Emir,
Let me give them a quick try and I'll update you.

Thanks,
Atita

On Tue, Oct 10, 2017 at 5:28 PM, Emir Arnautović <
[hidden email]> wrote:

> Hi Atita,
> I did not try it, but I think that following could work:
>
>
> #logging queries
> log4j.logger.org.apache.solr.handler.component.QueryComponent=WARN,slow
>
> log4j.appender.slow=org.apache.log4j.RollingFileAppender
> log4j.appender.slow.File=${solr.log}/slow.log
> log4j.appender.slow.layout=org.apache.log4j.EnhancedPatternLayout
> log4j.appender.slow.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS}
> %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
>
> If you want to log all queries, you can change level for query component
> to INFO.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 10 Oct 2017, at 13:35, Atita Arora <[hidden email]> wrote:
> >
> > Hi Emir,
> >
> > So I made few changes to the log4j config , I am able to redirect these
> > logs to another file as well.
> > But as these are the WARN logs so I doubt any logs enabled at WARN level
> > are going to be redirected here in this new log file.
> > So precisely , I am using Solr 6.1 (in cloud mode) & I have made few more
> > changes to the logging levels and components.
> > Please find my log4j at : *https://pastebin.com/uTLAiBE5
> > <https://pastebin.com/uTLAiBE5>*
> >
> > Any help on this will surely be appreciated.
> >
> > Thanks again.
> >
> > Atita
> >
> >
> > On Tue, Oct 10, 2017 at 1:39 PM, Emir Arnautović <
> > [hidden email]> wrote:
> >
> >> Hi Atita,
> >> You should definetely go with log4j configuration as anything else would
> >> be redoing what log4j can do. You already have
> slowQueryThresholdMillies to
> >> make slow queries log with WARN and you can configure log4j to put such
> >> logs (class + level) to a separate file.
> >> This seems like frequent question and not sure why putting logs to
> >> separate file is not a default configuration - maybe it would make
> things
> >> bit more complicated with logs view in admin console…
> >> If get stuck, let me know (+ Solr version) and I’ll play a bit and send
> >> you configs.
> >>
> >> HTH,
> >> Emir
> >> --
> >> Monitoring - Log Management - Alerting - Anomaly Detection
> >> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >>
> >>
> >>
> >>> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
> >>>
> >>> Hi ,
> >>>
> >>> I have a situation here where I am required to log the slow queries
> into
> >> a
> >>> seperate log file which then can be used for optimization purposes.
> >>> For now this log is aggregated into the mainstream log marking
> >>> [slow:......].
> >>> I looked into the code and the configuration and I am really clueless
> as
> >> to
> >>> how do I go about seperating the slow query logs as it needs another
> file
> >>> appender
> >>> to be created other than the one already present in the log4j.
> >>> If I create another appender I can do so by degregating through log
> >> levels
> >>> , so that moves all the WARN logs to another file (which is not what I
> am
> >>> looking for).
> >>> Also from the code prespective , I feel how about if I introduce
> another
> >>> config setting along with the slowQueryThresholdMillis value ,
> something
> >>> like
> >>>
> >>> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
> >>>
> >>>
> >>> where slowQueryLogFile and if present it logs into this file otherwise
> it
> >>> works on the already present along with
> >>>
> >>> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis",
> -1);
> >>>
> >>>
> >>> or should I tweak log4j ?
> >>> I am not sure if anyone has done that before or have any pointers to
> >> guide
> >>> me on this.
> >>> Please help.
> >>>
> >>> Thanks in advance,
> >>> Atita
> >>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Atita Arora
No luck for me , did you give it a try meantime ?
M not sure , if I may have missed something , my logs are completely gone
after this change.

Wondering whats wrong with them.

-Atita

On Tue, Oct 10, 2017 at 5:58 PM, Atita Arora <[hidden email]> wrote:

> Sure thanks Emir,
> Let me give them a quick try and I'll update you.
>
> Thanks,
> Atita
>
> On Tue, Oct 10, 2017 at 5:28 PM, Emir Arnautović <
> [hidden email]> wrote:
>
>> Hi Atita,
>> I did not try it, but I think that following could work:
>>
>>
>> #logging queries
>> log4j.logger.org.apache.solr.handler.component.QueryComponent=WARN,slow
>>
>> log4j.appender.slow=org.apache.log4j.RollingFileAppender
>> log4j.appender.slow.File=${solr.log}/slow.log
>> log4j.appender.slow.layout=org.apache.log4j.EnhancedPatternLayout
>> log4j.appender.slow.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS}
>> %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
>>
>> If you want to log all queries, you can change level for query component
>> to INFO.
>>
>> HTH,
>> Emir
>> --
>> Monitoring - Log Management - Alerting - Anomaly Detection
>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>
>>
>>
>> > On 10 Oct 2017, at 13:35, Atita Arora <[hidden email]> wrote:
>> >
>> > Hi Emir,
>> >
>> > So I made few changes to the log4j config , I am able to redirect these
>> > logs to another file as well.
>> > But as these are the WARN logs so I doubt any logs enabled at WARN level
>> > are going to be redirected here in this new log file.
>> > So precisely , I am using Solr 6.1 (in cloud mode) & I have made few
>> more
>> > changes to the logging levels and components.
>> > Please find my log4j at : *https://pastebin.com/uTLAiBE5
>> > <https://pastebin.com/uTLAiBE5>*
>> >
>> > Any help on this will surely be appreciated.
>> >
>> > Thanks again.
>> >
>> > Atita
>> >
>> >
>> > On Tue, Oct 10, 2017 at 1:39 PM, Emir Arnautović <
>> > [hidden email]> wrote:
>> >
>> >> Hi Atita,
>> >> You should definetely go with log4j configuration as anything else
>> would
>> >> be redoing what log4j can do. You already have
>> slowQueryThresholdMillies to
>> >> make slow queries log with WARN and you can configure log4j to put such
>> >> logs (class + level) to a separate file.
>> >> This seems like frequent question and not sure why putting logs to
>> >> separate file is not a default configuration - maybe it would make
>> things
>> >> bit more complicated with logs view in admin console…
>> >> If get stuck, let me know (+ Solr version) and I’ll play a bit and send
>> >> you configs.
>> >>
>> >> HTH,
>> >> Emir
>> >> --
>> >> Monitoring - Log Management - Alerting - Anomaly Detection
>> >> Solr & Elasticsearch Consulting Support Training -
>> http://sematext.com/
>> >>
>> >>
>> >>
>> >>> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
>> >>>
>> >>> Hi ,
>> >>>
>> >>> I have a situation here where I am required to log the slow queries
>> into
>> >> a
>> >>> seperate log file which then can be used for optimization purposes.
>> >>> For now this log is aggregated into the mainstream log marking
>> >>> [slow:......].
>> >>> I looked into the code and the configuration and I am really clueless
>> as
>> >> to
>> >>> how do I go about seperating the slow query logs as it needs another
>> file
>> >>> appender
>> >>> to be created other than the one already present in the log4j.
>> >>> If I create another appender I can do so by degregating through log
>> >> levels
>> >>> , so that moves all the WARN logs to another file (which is not what
>> I am
>> >>> looking for).
>> >>> Also from the code prespective , I feel how about if I introduce
>> another
>> >>> config setting along with the slowQueryThresholdMillis value ,
>> something
>> >>> like
>> >>>
>> >>> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
>> >>>
>> >>>
>> >>> where slowQueryLogFile and if present it logs into this file
>> otherwise it
>> >>> works on the already present along with
>> >>>
>> >>> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis",
>> -1);
>> >>>
>> >>>
>> >>> or should I tweak log4j ?
>> >>> I am not sure if anyone has done that before or have any pointers to
>> >> guide
>> >>> me on this.
>> >>> Please help.
>> >>>
>> >>> Thanks in advance,
>> >>> Atita
>> >>
>> >>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Emir Arnautović
Hi Atita,
I did not have time to try it out, but will try to do it over the weekend if you are still having troubles with it.

Regards,
Emir
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/



> On 10 Oct 2017, at 19:59, Atita Arora <[hidden email]> wrote:
>
> No luck for me , did you give it a try meantime ?
> M not sure , if I may have missed something , my logs are completely gone
> after this change.
>
> Wondering whats wrong with them.
>
> -Atita
>
> On Tue, Oct 10, 2017 at 5:58 PM, Atita Arora <[hidden email]> wrote:
>
>> Sure thanks Emir,
>> Let me give them a quick try and I'll update you.
>>
>> Thanks,
>> Atita
>>
>> On Tue, Oct 10, 2017 at 5:28 PM, Emir Arnautović <
>> [hidden email]> wrote:
>>
>>> Hi Atita,
>>> I did not try it, but I think that following could work:
>>>
>>>
>>> #logging queries
>>> log4j.logger.org.apache.solr.handler.component.QueryComponent=WARN,slow
>>>
>>> log4j.appender.slow=org.apache.log4j.RollingFileAppender
>>> log4j.appender.slow.File=${solr.log}/slow.log
>>> log4j.appender.slow.layout=org.apache.log4j.EnhancedPatternLayout
>>> log4j.appender.slow.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSS}
>>> %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
>>>
>>> If you want to log all queries, you can change level for query component
>>> to INFO.
>>>
>>> HTH,
>>> Emir
>>> --
>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>>>
>>>
>>>
>>>> On 10 Oct 2017, at 13:35, Atita Arora <[hidden email]> wrote:
>>>>
>>>> Hi Emir,
>>>>
>>>> So I made few changes to the log4j config , I am able to redirect these
>>>> logs to another file as well.
>>>> But as these are the WARN logs so I doubt any logs enabled at WARN level
>>>> are going to be redirected here in this new log file.
>>>> So precisely , I am using Solr 6.1 (in cloud mode) & I have made few
>>> more
>>>> changes to the logging levels and components.
>>>> Please find my log4j at : *https://pastebin.com/uTLAiBE5
>>>> <https://pastebin.com/uTLAiBE5>*
>>>>
>>>> Any help on this will surely be appreciated.
>>>>
>>>> Thanks again.
>>>>
>>>> Atita
>>>>
>>>>
>>>> On Tue, Oct 10, 2017 at 1:39 PM, Emir Arnautović <
>>>> [hidden email]> wrote:
>>>>
>>>>> Hi Atita,
>>>>> You should definetely go with log4j configuration as anything else
>>> would
>>>>> be redoing what log4j can do. You already have
>>> slowQueryThresholdMillies to
>>>>> make slow queries log with WARN and you can configure log4j to put such
>>>>> logs (class + level) to a separate file.
>>>>> This seems like frequent question and not sure why putting logs to
>>>>> separate file is not a default configuration - maybe it would make
>>> things
>>>>> bit more complicated with logs view in admin console…
>>>>> If get stuck, let me know (+ Solr version) and I’ll play a bit and send
>>>>> you configs.
>>>>>
>>>>> HTH,
>>>>> Emir
>>>>> --
>>>>> Monitoring - Log Management - Alerting - Anomaly Detection
>>>>> Solr & Elasticsearch Consulting Support Training -
>>> http://sematext.com/
>>>>>
>>>>>
>>>>>
>>>>>> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
>>>>>>
>>>>>> Hi ,
>>>>>>
>>>>>> I have a situation here where I am required to log the slow queries
>>> into
>>>>> a
>>>>>> seperate log file which then can be used for optimization purposes.
>>>>>> For now this log is aggregated into the mainstream log marking
>>>>>> [slow:......].
>>>>>> I looked into the code and the configuration and I am really clueless
>>> as
>>>>> to
>>>>>> how do I go about seperating the slow query logs as it needs another
>>> file
>>>>>> appender
>>>>>> to be created other than the one already present in the log4j.
>>>>>> If I create another appender I can do so by degregating through log
>>>>> levels
>>>>>> , so that moves all the WARN logs to another file (which is not what
>>> I am
>>>>>> looking for).
>>>>>> Also from the code prespective , I feel how about if I introduce
>>> another
>>>>>> config setting along with the slowQueryThresholdMillis value ,
>>> something
>>>>>> like
>>>>>>
>>>>>> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
>>>>>>
>>>>>>
>>>>>> where slowQueryLogFile and if present it logs into this file
>>> otherwise it
>>>>>> works on the already present along with
>>>>>>
>>>>>> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis",
>>> -1);
>>>>>>
>>>>>>
>>>>>> or should I tweak log4j ?
>>>>>> I am not sure if anyone has done that before or have any pointers to
>>>>> guide
>>>>>> me on this.
>>>>>> Please help.
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Atita
>>>>>
>>>>>
>>>
>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Atita Arora
Indeed , the trouble hasn't got over yet.
So we got
https://issues.apache.org/jira/browse/SOLR-11453

created meantime.

I'll look forward to your updates.

Thanks again ,
Atita

On Thu, Oct 12, 2017 at 2:08 PM, Emir Arnautović <
[hidden email]> wrote:

> Hi Atita,
> I did not have time to try it out, but will try to do it over the weekend
> if you are still having troubles with it.
>
> Regards,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
> > On 10 Oct 2017, at 19:59, Atita Arora <[hidden email]> wrote:
> >
> > No luck for me , did you give it a try meantime ?
> > M not sure , if I may have missed something , my logs are completely gone
> > after this change.
> >
> > Wondering whats wrong with them.
> >
> > -Atita
> >
> > On Tue, Oct 10, 2017 at 5:58 PM, Atita Arora <[hidden email]>
> wrote:
> >
> >> Sure thanks Emir,
> >> Let me give them a quick try and I'll update you.
> >>
> >> Thanks,
> >> Atita
> >>
> >> On Tue, Oct 10, 2017 at 5:28 PM, Emir Arnautović <
> >> [hidden email]> wrote:
> >>
> >>> Hi Atita,
> >>> I did not try it, but I think that following could work:
> >>>
> >>>
> >>> #logging queries
> >>> log4j.logger.org.apache.solr.handler.component.
> QueryComponent=WARN,slow
> >>>
> >>> log4j.appender.slow=org.apache.log4j.RollingFileAppender
> >>> log4j.appender.slow.File=${solr.log}/slow.log
> >>> log4j.appender.slow.layout=org.apache.log4j.EnhancedPatternLayout
> >>> log4j.appender.slow.layout.ConversionPattern=%d{yyyy-MM-dd
> HH:mm:ss.SSS}
> >>> %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
> >>>
> >>> If you want to log all queries, you can change level for query
> component
> >>> to INFO.
> >>>
> >>> HTH,
> >>> Emir
> >>> --
> >>> Monitoring - Log Management - Alerting - Anomaly Detection
> >>> Solr & Elasticsearch Consulting Support Training -
> http://sematext.com/
> >>>
> >>>
> >>>
> >>>> On 10 Oct 2017, at 13:35, Atita Arora <[hidden email]> wrote:
> >>>>
> >>>> Hi Emir,
> >>>>
> >>>> So I made few changes to the log4j config , I am able to redirect
> these
> >>>> logs to another file as well.
> >>>> But as these are the WARN logs so I doubt any logs enabled at WARN
> level
> >>>> are going to be redirected here in this new log file.
> >>>> So precisely , I am using Solr 6.1 (in cloud mode) & I have made few
> >>> more
> >>>> changes to the logging levels and components.
> >>>> Please find my log4j at : *https://pastebin.com/uTLAiBE5
> >>>> <https://pastebin.com/uTLAiBE5>*
> >>>>
> >>>> Any help on this will surely be appreciated.
> >>>>
> >>>> Thanks again.
> >>>>
> >>>> Atita
> >>>>
> >>>>
> >>>> On Tue, Oct 10, 2017 at 1:39 PM, Emir Arnautović <
> >>>> [hidden email]> wrote:
> >>>>
> >>>>> Hi Atita,
> >>>>> You should definetely go with log4j configuration as anything else
> >>> would
> >>>>> be redoing what log4j can do. You already have
> >>> slowQueryThresholdMillies to
> >>>>> make slow queries log with WARN and you can configure log4j to put
> such
> >>>>> logs (class + level) to a separate file.
> >>>>> This seems like frequent question and not sure why putting logs to
> >>>>> separate file is not a default configuration - maybe it would make
> >>> things
> >>>>> bit more complicated with logs view in admin console…
> >>>>> If get stuck, let me know (+ Solr version) and I’ll play a bit and
> send
> >>>>> you configs.
> >>>>>
> >>>>> HTH,
> >>>>> Emir
> >>>>> --
> >>>>> Monitoring - Log Management - Alerting - Anomaly Detection
> >>>>> Solr & Elasticsearch Consulting Support Training -
> >>> http://sematext.com/
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
> >>>>>>
> >>>>>> Hi ,
> >>>>>>
> >>>>>> I have a situation here where I am required to log the slow queries
> >>> into
> >>>>> a
> >>>>>> seperate log file which then can be used for optimization purposes.
> >>>>>> For now this log is aggregated into the mainstream log marking
> >>>>>> [slow:......].
> >>>>>> I looked into the code and the configuration and I am really
> clueless
> >>> as
> >>>>> to
> >>>>>> how do I go about seperating the slow query logs as it needs another
> >>> file
> >>>>>> appender
> >>>>>> to be created other than the one already present in the log4j.
> >>>>>> If I create another appender I can do so by degregating through log
> >>>>> levels
> >>>>>> , so that moves all the WARN logs to another file (which is not what
> >>> I am
> >>>>>> looking for).
> >>>>>> Also from the code prespective , I feel how about if I introduce
> >>> another
> >>>>>> config setting along with the slowQueryThresholdMillis value ,
> >>> something
> >>>>>> like
> >>>>>>
> >>>>>> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
> >>>>>>
> >>>>>>
> >>>>>> where slowQueryLogFile and if present it logs into this file
> >>> otherwise it
> >>>>>> works on the already present along with
> >>>>>>
> >>>>>> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis",
> >>> -1);
> >>>>>>
> >>>>>>
> >>>>>> or should I tweak log4j ?
> >>>>>> I am not sure if anyone has done that before or have any pointers to
> >>>>> guide
> >>>>>> me on this.
> >>>>>> Please help.
> >>>>>>
> >>>>>> Thanks in advance,
> >>>>>> Atita
> >>>>>
> >>>>>
> >>>
> >>>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Need help with Slow Query Logging

Walter Underwood
In reply to this post by Emir Arnautović
I would not do this in Solr.

Post process the log file to split them out. That allows you to change the definition of “slow” later, reprocess older files, etc.

Do log analysis with log analysis tools. Don’t try to push that too far up the chain into the production server.

wunder
Walter Underwood
[hidden email]
http://observer.wunderwood.org/  (my blog)


> On Oct 10, 2017, at 1:09 AM, Emir Arnautović <[hidden email]> wrote:
>
> Hi Atita,
> You should definetely go with log4j configuration as anything else would be redoing what log4j can do. You already have slowQueryThresholdMillies to make slow queries log with WARN and you can configure log4j to put such logs (class + level) to a separate file.
> This seems like frequent question and not sure why putting logs to separate file is not a default configuration - maybe it would make things bit more complicated with logs view in admin console…
> If get stuck, let me know (+ Solr version) and I’ll play a bit and send you configs.
>
> HTH,
> Emir
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
>
>> On 9 Oct 2017, at 16:27, Atita Arora <[hidden email]> wrote:
>>
>> Hi ,
>>
>> I have a situation here where I am required to log the slow queries into a
>> seperate log file which then can be used for optimization purposes.
>> For now this log is aggregated into the mainstream log marking
>> [slow:......].
>> I looked into the code and the configuration and I am really clueless as to
>> how do I go about seperating the slow query logs as it needs another file
>> appender
>> to be created other than the one already present in the log4j.
>> If I create another appender I can do so by degregating through log levels
>> , so that moves all the WARN logs to another file (which is not what I am
>> looking for).
>> Also from the code prespective , I feel how about if I introduce another
>> config setting along with the slowQueryThresholdMillis value , something
>> like
>>
>> slowQueryLogFile = get("query/slowQueryLogFile", logfilepath);
>>
>>
>> where slowQueryLogFile and if present it logs into this file otherwise it
>> works on the already present along with
>>
>> slowQueryThresholdMillis = getInt("query/slowQueryThresholdMillis", -1);
>>
>>
>> or should I tweak log4j ?
>> I am not sure if anyone has done that before or have any pointers to guide
>> me on this.
>> Please help.
>>
>> Thanks in advance,
>> Atita
>