Solr 7.4 and log4j2 JSONLayout

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Solr 7.4 and log4j2 JSONLayout

Michael Aleythe, Sternwald
Hey,

I'm trying to edit the log4j2 logging configuration for solr. The goal is to get a log file in json format. I configured the the JSONLayout for this purpose inside the rollingFile appender in the log4j2.xml. After this solr stops logging entirely. Solr.log file is empty. Only the solr-8983-console.log file contains 10 lines. The line "2018-09-06 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
My first guess was that the jackson-core and jackson-databind jars were missing, but that did not fix the problem.

Does anyone know where to find error-messages or exceptions that point me towards whats going wrong here?

This is my current log4j2 config file:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
  <Appenders>

    <Console name="STDOUT" target="SYSTEM_OUT">
      <PatternLayout>
        <Pattern>
          %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
        </Pattern>
      </PatternLayout>
    </Console>

    <RollingFile
        name="RollingFile"
        fileName="${sys:solr.log.dir}/solr.log"
        filePattern="${sys:solr.log.dir}/solr.log.%d{yyyy-MM-dd-hh}_%i.zip" >
      <JsonLayout compact="true" eventEol="true"/>
      <Policies>
        <OnStartupTriggeringPolicy />
        <SizeBasedTriggeringPolicy size="1 MB"/>
                    <TimeBasedTriggeringPolicy interval="1"/>
      </Policies>
      <DefaultRolloverStrategy max="10"/>
    </RollingFile>

    <RollingFile
        name="SlowFile"
        fileName="${sys:solr.log.dir}/solr_slow_requests.log"
        filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
      <PatternLayout>
        <Pattern>
          %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} %X{replica} %X{core}] %c{1.} %m%n
        </Pattern>
      </PatternLayout>
      <Policies>
        <OnStartupTriggeringPolicy />
        <SizeBasedTriggeringPolicy size="32 MB"/>
      </Policies>
      <DefaultRolloverStrategy max="10"/>
    </RollingFile>

  </Appenders>
  <Loggers>
    <Logger name="org.apache.hadoop" level="warn"/>
    <Logger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
    <Logger name="org.apache.zookeeper" level="warn"/>
    <Logger name="org.apache.solr.core.SolrCore.SlowRequest" level="info" additivity="false">
      <AppenderRef ref="SlowFile"/>
    <Root level="debug">
      <AppenderRef ref="RollingFile"/>
      <AppenderRef ref="STDOUT"/>
    </Root>
  </Loggers>
</Configuration>

Best regards
Michael Aleythe

Michael Aleythe
Team --(sr)^(ch)--
Java Entwickler | STERNWALD SYSTEMS GMBH

Fon +49 351 31 40 6010
Fax +49 351 31 40 6001

E-Mail [hidden email]
Skype [hidden email]
Web www.sternwald.com

STERNWALD SYSTEMS GMBH
Pohlandstraße 19, 01309 Dresden, Germany
Geschäftsführer Ard Meier
Registergericht Handelsregister Dresden, HRB 33480
UmSt-ID DE157125091

SUPPORT / HOTLINE
Fon +49 173 38 54 752
E-Mail [hidden email]
Web support.sternwald.net

STERNWALD Offices
Berlin | Dresden | Düsseldorf | Hamburg | Sofia | Würzburg

Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.4 and log4j2 JSONLayout

Varun Thacker-4
Hi,

Where did you add the jackson core and databind libs under? Maybe it's
conflicting with the JARs that Solr already ships?

Solr already comes with jackson dependencies

server/solr-webapp/webapp/WEB-INF/lib/ | grep jackson

jackson-annotations-2.9.5.jar

jackson-core-2.9.5.jar

jackson-core-asl-1.9.13.jar

jackson-databind-2.9.5.jar

jackson-dataformat-smile-2.9.5.jar

jackson-mapper-asl-1.9.13.jar

On Thu, Sep 6, 2018 at 6:46 AM Michael Aleythe, Sternwald <
[hidden email]> wrote:

> Hey,
>
> I'm trying to edit the log4j2 logging configuration for solr. The goal is
> to get a log file in json format. I configured the the JSONLayout for this
> purpose inside the rollingFile appender in the log4j2.xml. After this solr
> stops logging entirely. Solr.log file is empty. Only the
> solr-8983-console.log file contains 10 lines. The line "2018-09-06
> 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
> My first guess was that the jackson-core and jackson-databind jars were
> missing, but that did not fix the problem.
>
> Does anyone know where to find error-messages or exceptions that point me
> towards whats going wrong here?
>
> This is my current log4j2 config file:
>
> <?xml version="1.0" encoding="UTF-8"?>
> <Configuration>
>   <Appenders>
>
>     <Console name="STDOUT" target="SYSTEM_OUT">
>       <PatternLayout>
>         <Pattern>
>           %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard}
> %X{replica} %X{core}] %c{1.} %m%n
>         </Pattern>
>       </PatternLayout>
>     </Console>
>
>     <RollingFile
>         name="RollingFile"
>         fileName="${sys:solr.log.dir}/solr.log"
>
> filePattern="${sys:solr.log.dir}/solr.log.%d{yyyy-MM-dd-hh}_%i.zip" >
>       <JsonLayout compact="true" eventEol="true"/>
>       <Policies>
>         <OnStartupTriggeringPolicy />
>         <SizeBasedTriggeringPolicy size="1 MB"/>
>                     <TimeBasedTriggeringPolicy interval="1"/>
>       </Policies>
>       <DefaultRolloverStrategy max="10"/>
>     </RollingFile>
>
>     <RollingFile
>         name="SlowFile"
>         fileName="${sys:solr.log.dir}/solr_slow_requests.log"
>         filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
>       <PatternLayout>
>         <Pattern>
>           %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard}
> %X{replica} %X{core}] %c{1.} %m%n
>         </Pattern>
>       </PatternLayout>
>       <Policies>
>         <OnStartupTriggeringPolicy />
>         <SizeBasedTriggeringPolicy size="32 MB"/>
>       </Policies>
>       <DefaultRolloverStrategy max="10"/>
>     </RollingFile>
>
>   </Appenders>
>   <Loggers>
>     <Logger name="org.apache.hadoop" level="warn"/>
>     <Logger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
>     <Logger name="org.apache.zookeeper" level="warn"/>
>     <Logger name="org.apache.solr.core.SolrCore.SlowRequest" level="info"
> additivity="false">
>       <AppenderRef ref="SlowFile"/>
>     <Root level="debug">
>       <AppenderRef ref="RollingFile"/>
>       <AppenderRef ref="STDOUT"/>
>     </Root>
>   </Loggers>
> </Configuration>
>
> Best regards
> Michael Aleythe
>
> Michael Aleythe
> Team --(sr)^(ch)--
> Java Entwickler | STERNWALD SYSTEMS GMBH
>
> Fon +49 351 31 40 6010
> Fax +49 351 31 40 6001
>
> E-Mail [hidden email]
> Skype [hidden email]
> Web www.sternwald.com
>
> STERNWALD SYSTEMS GMBH
> Pohlandstraße 19, 01309 Dresden, Germany
> Geschäftsführer Ard Meier
> Registergericht Handelsregister Dresden, HRB 33480
> UmSt-ID DE157125091
>
> SUPPORT / HOTLINE
> Fon +49 173 38 54 752
> E-Mail [hidden email]
> Web support.sternwald.net
>
> STERNWALD Offices
> Berlin | Dresden | Düsseldorf | Hamburg | Sofia | Würzburg
>
>
Reply | Threaded
Open this post in threaded view
|

AW: Solr 7.4 and log4j2 JSONLayout

Michael Aleythe, Sternwald
Hey,

I tried solr/server/lib/ext and solr/server/lib. I also tried without them but it doesn't change anything.

Best regards
Michael

-----Ursprüngliche Nachricht-----
Von: Varun Thacker <[hidden email]>
Gesendet: Donnerstag, 6. September 2018 16:23
An: [hidden email]
Betreff: Re: Solr 7.4 and log4j2 JSONLayout

Hi,

Where did you add the jackson core and databind libs under? Maybe it's conflicting with the JARs that Solr already ships?

Solr already comes with jackson dependencies

server/solr-webapp/webapp/WEB-INF/lib/ | grep jackson

jackson-annotations-2.9.5.jar

jackson-core-2.9.5.jar

jackson-core-asl-1.9.13.jar

jackson-databind-2.9.5.jar

jackson-dataformat-smile-2.9.5.jar

jackson-mapper-asl-1.9.13.jar

On Thu, Sep 6, 2018 at 6:46 AM Michael Aleythe, Sternwald < [hidden email]> wrote:

> Hey,
>
> I'm trying to edit the log4j2 logging configuration for solr. The goal
> is to get a log file in json format. I configured the the JSONLayout
> for this purpose inside the rollingFile appender in the log4j2.xml.
> After this solr stops logging entirely. Solr.log file is empty. Only
> the solr-8983-console.log file contains 10 lines. The line "2018-09-06
> 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
> My first guess was that the jackson-core and jackson-databind jars
> were missing, but that did not fix the problem.
>
> Does anyone know where to find error-messages or exceptions that point
> me towards whats going wrong here?
>
> This is my current log4j2 config file:
>
> <?xml version="1.0" encoding="UTF-8"?> <Configuration>
>   <Appenders>
>
>     <Console name="STDOUT" target="SYSTEM_OUT">
>       <PatternLayout>
>         <Pattern>
>           %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection}
> %X{shard} %X{replica} %X{core}] %c{1.} %m%n
>         </Pattern>
>       </PatternLayout>
>     </Console>
>
>     <RollingFile
>         name="RollingFile"
>         fileName="${sys:solr.log.dir}/solr.log"
>
> filePattern="${sys:solr.log.dir}/solr.log.%d{yyyy-MM-dd-hh}_%i.zip" >
>       <JsonLayout compact="true" eventEol="true"/>
>       <Policies>
>         <OnStartupTriggeringPolicy />
>         <SizeBasedTriggeringPolicy size="1 MB"/>
>                     <TimeBasedTriggeringPolicy interval="1"/>
>       </Policies>
>       <DefaultRolloverStrategy max="10"/>
>     </RollingFile>
>
>     <RollingFile
>         name="SlowFile"
>         fileName="${sys:solr.log.dir}/solr_slow_requests.log"
>         filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
>       <PatternLayout>
>         <Pattern>
>           %d{yyyy-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection}
> %X{shard} %X{replica} %X{core}] %c{1.} %m%n
>         </Pattern>
>       </PatternLayout>
>       <Policies>
>         <OnStartupTriggeringPolicy />
>         <SizeBasedTriggeringPolicy size="32 MB"/>
>       </Policies>
>       <DefaultRolloverStrategy max="10"/>
>     </RollingFile>
>
>   </Appenders>
>   <Loggers>
>     <Logger name="org.apache.hadoop" level="warn"/>
>     <Logger name="org.apache.solr.update.LoggingInfoStream" level="off"/>
>     <Logger name="org.apache.zookeeper" level="warn"/>
>     <Logger name="org.apache.solr.core.SolrCore.SlowRequest" level="info"
> additivity="false">
>       <AppenderRef ref="SlowFile"/>
>     <Root level="debug">
>       <AppenderRef ref="RollingFile"/>
>       <AppenderRef ref="STDOUT"/>
>     </Root>
>   </Loggers>
> </Configuration>
>
> Best regards
> Michael Aleythe
>
> Michael Aleythe
> Team --(sr)^(ch)--
> Java Entwickler | STERNWALD SYSTEMS GMBH
>
> Fon +49 351 31 40 6010
> Fax +49 351 31 40 6001
>
> E-Mail [hidden email]
> Skype [hidden email]
> Web www.sternwald.com
>
> STERNWALD SYSTEMS GMBH
> Pohlandstraße 19, 01309 Dresden, Germany Geschäftsführer Ard Meier
> Registergericht Handelsregister Dresden, HRB 33480 UmSt-ID DE157125091
>
> SUPPORT / HOTLINE
> Fon +49 173 38 54 752
> E-Mail [hidden email]
> Web support.sternwald.net
>
> STERNWALD Offices
> Berlin | Dresden | Düsseldorf | Hamburg | Sofia | Würzburg
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Solr 7.4 and log4j2 JSONLayout

Shawn Heisey-2
In reply to this post by Michael Aleythe, Sternwald
On 9/6/2018 7:46 AM, Michael Aleythe, Sternwald wrote:
> I'm trying to edit the log4j2 logging configuration for solr. The goal is to get a log file in json format. I configured the the JSONLayout for this purpose inside the rollingFile appender in the log4j2.xml. After this solr stops logging entirely. Solr.log file is empty. Only the solr-8983-console.log file contains 10 lines. The line "2018-09-06 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
> My first guess was that the jackson-core and jackson-databind jars were missing, but that did not fix the problem.

As Varun said, jackson is already included in Solr.  You won't need to
add any jars for that.

> Does anyone know where to find error-messages or exceptions that point me towards whats going wrong here?

Start Solr in the foreground, with the -f option.  This will cause Solr
to log to the console.  When Solr is started in the background, it
suppresses console logging.

I see that you have changed the logfile rollover size to 1MB.  If your
Solr server sees much traffic, this is going to result in an extremely
fast rollover, which may mean that you lose access to logged events VERY
quickly.  This will especially be the case with JSON logging -- each
event will take up a lot more space.

Thanks,
Shawn

Reply | Threaded
Open this post in threaded view
|

AW: Solr 7.4 and log4j2 JSONLayout

Michael Aleythe, Sternwald
Hey Shawn,

i removed the additional jars and tried the foreground option with the following result:

2018-09-10 11:41:03.174:INFO::main: Logging initialized @616ms to org.eclipse.jetty.util.log.StdErrLog
2018-09-10 11:41:03.313:INFO:oejs.Server:main: jetty-9.4.10.v20180503; built: 2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 1.8.0_45-b14
2018-09-10 11:41:03.338:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///usr/local/dir/solr-7.4.0/server/contexts/] at interval 0
2018-09-10 11:41:03.578:INFO:oejw.StandardDescriptorProcessor:main: NO JSP Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2018-09-10 11:41:03.586:INFO:oejs.session:main: DefaultSessionIdManager workerName=node0
2018-09-10 11:41:03.586:INFO:oejs.session:main: No SessionScavenger set, using defaults
2018-09-10 11:41:03.588:INFO:oejs.session:main: node0 Scavenging every 600000ms
2018-09-10 11:41:05.091:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@7c7a06ec{/solr,file:///usr/local/dir/solr-7.4.0/server/solr-webapp/webapp/,AVAILABLE}{file:///usr/local/dir/solr-7.4.0/server//solr-webapp/webapp}
2018-09-10 11:41:05.133:INFO:oejs.AbstractConnector:main: Started ServerConnector@10650953{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2018-09-10 11:41:05.133:INFO:oejs.Server:main: Started @2575ms

There is absolutely no clue about whats going on.

Regarding the rollover size. I was just playing around to see if the rollover works as I expected.

Thanks,
Michael

-----Ursprüngliche Nachricht-----
Von: Shawn Heisey <[hidden email]>
Gesendet: Freitag, 7. September 2018 17:58
An: [hidden email]
Betreff: Re: Solr 7.4 and log4j2 JSONLayout

On 9/6/2018 7:46 AM, Michael Aleythe, Sternwald wrote:
> I'm trying to edit the log4j2 logging configuration for solr. The goal is to get a log file in json format. I configured the the JSONLayout for this purpose inside the rollingFile appender in the log4j2.xml. After this solr stops logging entirely. Solr.log file is empty. Only the solr-8983-console.log file contains 10 lines. The line "2018-09-06 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
> My first guess was that the jackson-core and jackson-databind jars were missing, but that did not fix the problem.

As Varun said, jackson is already included in Solr.  You won't need to add any jars for that.

> Does anyone know where to find error-messages or exceptions that point me towards whats going wrong here?

Start Solr in the foreground, with the -f option.  This will cause Solr to log to the console.  When Solr is started in the background, it suppresses console logging.

I see that you have changed the logfile rollover size to 1MB.  If your Solr server sees much traffic, this is going to result in an extremely fast rollover, which may mean that you lose access to logged events VERY quickly.  This will especially be the case with JSON logging -- each event will take up a lot more space.

Thanks,
Shawn