Sunil's Notes: logging

Apache Kafka has a KafkaLog4jAppender that you can use for redirecting your Log4j log to Kafka topic. I wanted to try it out so i used following steps, you can download sample project from here First i created a simple standalone java program that use Log4j like this. As you can see this is like any other normal Java program that uses Log4j. Then in the log4j.properties file i added line 12 to 17 for using KafkaLog4jAppender, on line 13, value of brokerList property points to the Kafka server and line 14 value of topic points to the Kafka topic name to which logs should go. Now before running this program make sure that you actually have topic named kafkalogger, if not you can create using this command


bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkalogger

You can verify if you have topic named kafkalogger by executing following command


bin/kafka-topics.sh --list --zookeeper localhost:2181

Also you can run kafka console consumer that reads messages from Kafka and prints them to console, using following command


bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafkalogger

Now when you run your java program you should see messages on console like this

Sometimes you might want to configure your logs so that it adds some context specific attributes on every line. Ex. you might want to print name of the logged in user in the log statement. So that if you want to see what went wrong for say user John you can find all the log statements for John by grep and analyze the problem Apache Log4j has MDC.java class that can be used for this type of use case. Basic idea is you set a Map of parameters on the current thread and that would be available to all the methods downstream. I wanted to try this feature out so i used the following steps.

First call MDC.put("USER","Sunil") method to set USER context variable at current thread level


package com.test.mq;

import org.apache.log4j.Logger;
import org.apache.log4j.MDC;
public class HelloMDC {
    public static void main(String[] argv){
        Logger logger = Logger.getLogger(HelloMDC.class);
        MDC.put("USER","Sunil");
        logger.debug("Sample debug message");
    }
}

Then you can configure the message pattern layout to include the USER variable like %X{USER} at the start of the message


<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
    <appender name="console" class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.out"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%X{USER} - %-5p %c{1} - %m%n"/>
        </layout>
    </appender>
    <root>
        <priority value ="debug" />
        <appender-ref ref="console" />
    </root>
</log4j:configuration>

Now when the code executes you can see the user name printed at the start of the message like this


Sunil - DEBUG HelloSender - Sample debug message

Sunil's Notes

How to use KafkaLog4jAppender for sending Log4j logs to kafka

Using MDC for setting context variables in Log4J