Inbound Thinking

At HubSpot we use log4j with all of our Java projects. It’s an excellent logging tool and for the most part makes life with logs much easier. Recently I wanted to convert a bunch of projects from just logging to file to send emails when errors were logged. Luckily log4j has an email appender (SMTPAppender) that does exactly this. You just set the threshold of event (Error in my case) and the buffer size for previous events. Say you want to get emailed the last 100 lines logged before an error, just set up an appender with a threshold of Error and a buffer size of 100, sounds easy.

The interesting part (for me) was that I wanted to differentiate emails sent on my Production servers from my QA servers. Since I always deploy the exact same package to Production and QA I couldn’t change the appender definition. The answer was to use dynamic appender configuration.

Basically, any log4j configuration can read system-level variables. You can pull the hostname of the server, or even the IP address if you want, and use that value in the appender configuration. In my case I opted to use a system variable to specify the operating environment (Production or QA in my case). Here’s an example of my appender configuration:

<!-- Email Appender sends on ERROR level -->
<appender name="email" class="org.apache.log4j.net.SMTPAppender">
<param name="subject" value="[LOG] ${app.log.environment} PasswordReset” />
<param name=”to” value=”logging@server.com” />
<param name=”from” value=”noreply@server.com” />
<param name=”SMTPHost” value=”mail.server.com” />
<param name=”bufferSize” value=”512″ />
<param name=”threshold” value=”ERROR” />
<layout class=”org.apache.log4j.PatternLayout”>
<param name=”ConversionPattern” value=”%d [%t] %-5p %c - %m%n” />
</layout>
</appender>

If you are using Tomcat to run the Java application, set the system variable in the  bin/catalina.sh file. Just add or update the JAVA_OPTS variable like this:

JAVA_OPTS="-Dapp.log.environment=QA"

There turned out to be a couple of tricky bits before I was able to get this working. The first was that the use of system variables to dynamically configure Appenders doesn’t appear anywhere in the log4j documentation. I managed to figure it out with excessive Googling and putting together pieces from all the sources. The second tricky part is that the Appender threshold doesn’t really matter. In practice, the Email Appender will only fire off a message when an event of Error-level or higher (Fatal)  is logged. For instance, you can not set an Email Appender to send messages on Info or Debug level events.

Overall, log4j dynamic appender configuration is powerful and super useful, hopefully this post can help others avoid the trouble I had getting it set up.