Modify the log directory for dih

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Modify the log directory for dih

lala
Hi,
Is there a way to set the log directory for a dih request and the log file
name?
thanks in advance...



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

Jan Høydahl / Cominvent
I'm not sure which log file you are referring to. You can configure where Solr puts its log files through the SOLR_LOGS_DIR environment setting, see
https://lucene.apache.org/solr/guide/7_4/configuring-logging.html#permanent-logging-settings <https://lucene.apache.org/solr/guide/7_4/configuring-logging.html#permanent-logging-settings>

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 2. okt. 2018 kl. 10:25 skrev lala <[hidden email]>:
>
> Hi,
> Is there a way to set the log directory for a dih request and the log file
> name?
> thanks in advance...
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

lala
I know tha Solr logs the dih operations (& most of other operations) in
server\logs\solr.log file. What I want is to configure the dih requests to
be logged in another path, with another name if it's possible.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

Shawn Heisey-2
On 10/2/2018 3:33 AM, lala wrote:
> I know tha Solr logs the dih operations (& most of other operations) in
> server\logs\solr.log file. What I want is to configure the dih requests to
> be logged in another path, with another name if it's possible.

DIH doesn't make its own logfile.  Just like the rest of Solr, it uses
the slf4j logging framework, which the stock Solr package binds to log4j
(log4j2 as of version 7.4.0, log4j 1.2.x before that) ... there is a
log4j configuration file included with Solr that controls logging.

With a change to the log4j configuration file, you can direct all logs
created by the DIH classes to a separate file, no code changes needed.

The actual requests to the dataimport handler (usually /dataimport) are
logged by the SolrCore class, not the DIH classes, so that part is
nearly impossible to separate from requests to other handlers.  The SQL
requests that the SQL entity makes are not logged by default, but if you
change the level of the SQL entity class to DEBUG instead of INFO, they
do get logged.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

lala
Thanks a lot Shawn for your reply,

For what you said:

Shawn Heisey-2 wrote
> With a change to the log4j configuration file, you can direct all logs
> created by the DIH classes to a separate file, no code changes needed.

Since I'm a newbee regarding log4j, Can you please give me an example about
how to change this configuration file for DIH?

Thanks in advance...



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

Shawn Heisey-2
On 10/2/2018 10:49 PM, lala wrote:
> Shawn Heisey-2 wrote
>> With a change to the log4j configuration file, you can direct all logs
>> created by the DIH classes to a separate file, no code changes needed.
> Since I'm a newbee regarding log4j, Can you please give me an example about
> how to change this configuration file for DIH?

What version of Solr do you have, what OS is it running on, and how did
you install/start it?

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

lala
Hi,
I am using:

Solr: 7.4
OS: windows7
I start solr using a service on startup.

Additional info: I am developing a web application that uses solr as search
engine, I use DIH to index folders in solr using the
FileListEntityProcessor. What I need is logging each index operation in a
file that I can reach & read to be able to detect failed index files in the
folder.

Thanks.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

Shawn Heisey-2
On 10/4/2018 12:30 AM, lala wrote:
> Hi,
> I am using:
>
> Solr: 7.4
> OS: windows7
> I start solr using a service on startup.

In that case, I really have no idea where anything is on your system.

There is no service installation from the Solr project for Windows --
either you obtained that from somewhere else, or it's something written
in-house.  Either way, you would need to talk to whoever created that
service installation for help locating files on your setup.

In general, you need to find the log4j2.xml file that is controlling
your logging configuration and modify it.  It contains a sample of how
to log something to a separate file -- the slow query log.  That example
redirects a specific logger name (which is similar to a full qualified
class name and in most cases *is* the class name) to a different logfile.

Version 7.4 has a bug when running on Windows that causes a lot of
problems specific to logging.

https://issues.apache.org/jira/browse/SOLR-12538

That problem has been fixed in the 7.5 release.  You can also fix it by
editing the solr.cmd script manually.

> Additional info: I am developing a web application that uses solr as search
> engine, I use DIH to index folders in solr using the
> FileListEntityProcessor. What I need is logging each index operation in a
> file that I can reach & read to be able to detect failed index files in the
> folder.

The FileListEntityProcessor class has absolutely no logging in it.  If
you require that immediately, you would need to add logging commands to
the source code and recompile Solr yourself to produce a package with
your change.  With an enhancement issue in Jira, we can review what
logging is suitable for the class, and probably make it work like
SQLEntityProcessor in that regard.  If that's done the way I think it
should be, then you could add config in log4j2.xml to could enable DEBUG
level logging for that class specifically and write its logs to a
separate logfile.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

Charlie Hull-3
On 04/10/2018 16:35, Shawn Heisey wrote:

> On 10/4/2018 12:30 AM, lala wrote:
>> Hi,
>> I am using:
>>
>> Solr: 7.4
>> OS: windows7
>> I start solr using a service on startup.
>
> In that case, I really have no idea where anything is on your system.
>
> There is no service installation from the Solr project for Windows --
> either you obtained that from somewhere else, or it's something written
> in-house.  Either way, you would need to talk to whoever created that
> service installation for help locating files on your setup.

We usually use NSSM for service-ifying Solr on Windows, I'd recommend
you consider that. Also, bear in mind that a Windows Service can't
output to stdout or stderr so some messages simply won't go anywhere -
but the NSSM documentation is helpful.

Charlie

>
> In general, you need to find the log4j2.xml file that is controlling
> your logging configuration and modify it.  It contains a sample of how
> to log something to a separate file -- the slow query log.  That example
> redirects a specific logger name (which is similar to a full qualified
> class name and in most cases *is* the class name) to a different logfile.
>
> Version 7.4 has a bug when running on Windows that causes a lot of
> problems specific to logging.
>
> https://issues.apache.org/jira/browse/SOLR-12538
>
> That problem has been fixed in the 7.5 release.  You can also fix it by
> editing the solr.cmd script manually.
>
>> Additional info: I am developing a web application that uses solr as
>> search
>> engine, I use DIH to index folders in solr using the
>> FileListEntityProcessor. What I need is logging each index operation in a
>> file that I can reach & read to be able to detect failed index files
>> in the
>> folder.
>
> The FileListEntityProcessor class has absolutely no logging in it.  If
> you require that immediately, you would need to add logging commands to
> the source code and recompile Solr yourself to produce a package with
> your change.  With an enhancement issue in Jira, we can review what
> logging is suitable for the class, and probably make it work like
> SQLEntityProcessor in that regard.  If that's done the way I think it
> should be, then you could add config in log4j2.xml to could enable DEBUG
> level logging for that class specifically and write its logs to a
> separate logfile.
>
> Thanks,
> Shawn
>


--
Charlie Hull
Flax - Open Source Enterprise Search

tel/fax: +44 (0)8700 118334
mobile:  +44 (0)7767 825828
web: www.flax.co.uk
Reply | Threaded
Open this post in threaded view
|

Re: Modify the log directory for dih

lala
In reply to this post by Shawn Heisey-2
Hi,
I installed solr as a service onwindows using nssm.
The log4j2.xml file resides in solr\server\resources.

I managed to direct logging of the class "LogUpdateProcessorFactory" to a
new location (a sub-directory in logs).
My problem now is:
- I need to create a new log file for each dih request I send to solr.
- I need more information about indexing process: the file beeing indexed,
whether it succeded or not, the cause af failures if exists...

This is what I've added to the xml configuration file log4j2.xml:
In appenders:
<Routing name="DihLog">
<Routes pattern="$${ctx:ROUTINGKEY}>
<Route key="$${ctx:ROUTINGKEY}">
<RollingFile name="DihFile"
fileName="${solr.log.dir}\dih\dih_${date:yyyy-MM-dd-HH-mm}.log"
filePattern="${solr.log.dir}\dih\dih_%d{yyyy-MM-dd-HH-mm}.log" >
<patternLayout><pattern>[%p] %c{1.} %m%n </pattern></PatternLayout>
<Policies><TimeBasedTriggeringPolicy /></Policies>
</RollingFile>
</Route>
</Routes>
</Routing>


And In Loggers:
<Logger name="org.apache.solr.update.processor.LogUpdateProcessorFactory"
level="all" additivity="false">
<AppenderRef ref="DihLog" />
</Logger>

Now, a dih file named by date is created & the messages of the class
LogUpdateProcessorFactory are logged into it, but if tow requests are sent
in a very short interval of time, they both are logged in the same file.
I appreciate any help.



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html