How to use KafkaLog4jAppender for sending Log4j logs to kafka

Apache Kafka has a KafkaLog4jAppender that you can use for redirecting your Log4j log to Kafka topic. I wanted to try it out so i used following steps, you can download sample project from here First i created a simple standalone java program that use Log4j like this. As you can see this is like any other normal Java program that uses Log4j. Then in the log4j.properties file i added line 12 to 17 for using KafkaLog4jAppender, on line 13, value of brokerList property points to the Kafka server and line 14 value of topic points to the Kafka topic name to which logs should go. Now before running this program make sure that you actually have topic named kafkalogger, if not you can create using this command

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkalogger
You can verify if you have topic named kafkalogger by executing following command

bin/kafka-topics.sh --list --zookeeper localhost:2181
Also you can run kafka console consumer that reads messages from Kafka and prints them to console, using following command

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafkalogger
Now when you run your java program you should see messages on console like this

15 comments:

Anonymous said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Kafka, kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training on in Apache Kafka. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Nitesh Kumar
MaxMunus
E-mail: nitesh@maxmunus.com
Skype id: nitesh_maxmunus
Ph:(+91) 8553912023
http://www.maxmunus.com/


Anonymous said...

Hello Sunil,

I downloaded your project and on running it I see a warning,

No such property [level] in org.apache.kafka.log4jappender.KafkaLog4jAppender, so I just commented it out.

log4j.appender.kafka.brokerList=localhost:9092
log4j.appender.kafka.topic=kafkalogger

I changed the above two properties to point to a remote Kafka broker and I dont get any logs. I dont get any error messages either. All I see is Eclipse console showing the messages. Shouldn't I be able to see the same messages on remote Kafka topic as well? Thanks.

jeisonmp said...

Dont work for me.

Kshetra Media House said...

This is my first time I have visited here. logo designing services in Hyderabad
I found a lot of interesting stuff in your blog. keep up the impressive work.
logo designing services
logo designing in hyderabad
logo designers in hyderabad

Ideal Designs India said...

Nice Blog
At Ideal designs, we believe that design should not just look good. It should also work well; it should be memorable and it should connect with the end-user. The success of many a business today depends on how vital a role ‘good’ design plays in their business strategy. Design is often the first thing that the consumer comes in contact with, before you can meet the company representative, you see the logo… and ‘good’ or ‘effective’ design has already started playing a vital role.Logo Design Hyderabad, Logo Design in Hyderabad, Brand Logo Design in Hyderabad, Corporate Logo Design in Hyderabad, Logo Design at Hyderabad.

Lakshmi said...

Nice Blog
We are a bunch of focused individuals with strong knowledge base, collaborative insights and cutting-edge ideas. We have been working with clients across various industry sectors, helping their brands perform consistently in dynamic market scenarios. Our team of experienced advertising and design professionals embraces all methods of communication to trigger the best results for our clients. Our creative methodologies are designed to spark original ideas and new ones.

Lakshmi said...

We are a bunch of focused individuals with strong knowledge base, collaborative insights and cutting-edge ideas. We have been working with clients across various industry sectors, helping their brands perform consistently in dynamic market scenarios. Our team of experienced advertising and design professionals embraces all methods of communication to trigger the best results for our clients. Our creative methodologies are designed to spark original ideas and new ones.

raju said...

Nice Blog

We are a bunch of focused individuals with strong knowledge base, collaborative insights and cutting-edge ideas. We have been working with clients across various industry sectors, helping their brands perform consistently in dynamic market scenarios. Our team of experienced advertising and design professionals embraces all methods of communication to trigger the best results for our clients. Our creative methodologies are designed to spark original ideas and new ones.

visit : http://idealbranding.in/

dipanshu sharma said...

Excellent web site you have here.. It’s hard to find good quality writing like yours these days. I truly appreciate individuals like you!
Debt Consolidation lawyer in Ontario

dipanshu sharma said...
This comment has been removed by the author.
dipanshu sharma said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
Study Permit Visa for Canada

dipanshu sharma said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
Notary Public Lawyer in Kitchener

Sunny said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging. Body building Product Review

ideal said...

Nice Blog

Ideal Branding will help you Creating a unique brand identity is absolutely crucial especially if you are competing in a market that is saturated with other businesses selling the same product or service. It is increasingly vital to create brand differentiation right from the very beginning to stand-out in the market. By authentically expressing their brand identity and adopting a systematic brand-building approach, we help SMEs and start-ups build their own valuable brands for stronger sales and better business performance.

raju said...

Great content it was really attractive


Construction companies often request strong, masculine logos with bold colours. Whether you’re looking for something similar or to break from the norm, we make it easy to get a design that’s right for you. Brochures are perfect for promoting and providing essential information about the business. A clean, professional, and high-impact designed brochure sets you apart from your competition and reinforces the value your company offers. Ideal Design makes brochures Design different. Unique designs and strategic objectives turn prospective leads into clients or customers.


http://idealbranding.in/

idealbranding05@gmail.com