How to use KafkaLog4jAppender for sending Log4j logs to kafka

Apache Kafka has a KafkaLog4jAppender that you can use for redirecting your Log4j log to Kafka topic. I wanted to try it out so i used following steps, you can download sample project from here First i created a simple standalone java program that use Log4j like this. As you can see this is like any other normal Java program that uses Log4j. Then in the log4j.properties file i added line 12 to 17 for using KafkaLog4jAppender, on line 13, value of brokerList property points to the Kafka server and line 14 value of topic points to the Kafka topic name to which logs should go. Now before running this program make sure that you actually have topic named kafkalogger, if not you can create using this command

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkalogger
You can verify if you have topic named kafkalogger by executing following command

bin/kafka-topics.sh --list --zookeeper localhost:2181
Also you can run kafka console consumer that reads messages from Kafka and prints them to console, using following command

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafkalogger
Now when you run your java program you should see messages on console like this

15 comments:

  1. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Kafka, kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on in Apache Kafka. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Nitesh Kumar
    MaxMunus
    E-mail: nitesh@maxmunus.com
    Skype id: nitesh_maxmunus
    Ph:(+91) 8553912023
    http://www.maxmunus.com/


    ReplyDelete
  2. Hello Sunil,

    I downloaded your project and on running it I see a warning,

    No such property [level] in org.apache.kafka.log4jappender.KafkaLog4jAppender, so I just commented it out.

    log4j.appender.kafka.brokerList=localhost:9092
    log4j.appender.kafka.topic=kafkalogger

    I changed the above two properties to point to a remote Kafka broker and I dont get any logs. I dont get any error messages either. All I see is Eclipse console showing the messages. Shouldn't I be able to see the same messages on remote Kafka topic as well? Thanks.

    ReplyDelete
  3. This is my first time I have visited here. logo designing services in Hyderabad
    I found a lot of interesting stuff in your blog. keep up the impressive work.
    logo designing services
    logo designing in hyderabad
    logo designers in hyderabad

    ReplyDelete
  4. Nice Blog
    At Ideal designs, we believe that design should not just look good. It should also work well; it should be memorable and it should connect with the end-user. The success of many a business today depends on how vital a role ‘good’ design plays in their business strategy. Design is often the first thing that the consumer comes in contact with, before you can meet the company representative, you see the logo… and ‘good’ or ‘effective’ design has already started playing a vital role.Logo Design Hyderabad, Logo Design in Hyderabad, Brand Logo Design in Hyderabad, Corporate Logo Design in Hyderabad, Logo Design at Hyderabad.

    ReplyDelete
  5. Nice Blog
    We are a bunch of focused individuals with strong knowledge base, collaborative insights and cutting-edge ideas. We have been working with clients across various industry sectors, helping their brands perform consistently in dynamic market scenarios. Our team of experienced advertising and design professionals embraces all methods of communication to trigger the best results for our clients. Our creative methodologies are designed to spark original ideas and new ones.

    ReplyDelete
  6. We are a bunch of focused individuals with strong knowledge base, collaborative insights and cutting-edge ideas. We have been working with clients across various industry sectors, helping their brands perform consistently in dynamic market scenarios. Our team of experienced advertising and design professionals embraces all methods of communication to trigger the best results for our clients. Our creative methodologies are designed to spark original ideas and new ones.

    ReplyDelete
  7. Nice Blog

    We are a bunch of focused individuals with strong knowledge base, collaborative insights and cutting-edge ideas. We have been working with clients across various industry sectors, helping their brands perform consistently in dynamic market scenarios. Our team of experienced advertising and design professionals embraces all methods of communication to trigger the best results for our clients. Our creative methodologies are designed to spark original ideas and new ones.

    visit : http://idealbranding.in/

    ReplyDelete
  8. Excellent web site you have here.. It’s hard to find good quality writing like yours these days. I truly appreciate individuals like you!
    Debt Consolidation lawyer in Ontario

    ReplyDelete
  9. This comment has been removed by the author.

    ReplyDelete
  10. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Study Permit Visa for Canada

    ReplyDelete
  11. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    Notary Public Lawyer in Kitchener

    ReplyDelete
  12. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging. Body building Product Review

    ReplyDelete
  13. Nice Blog

    Ideal Branding will help you Creating a unique brand identity is absolutely crucial especially if you are competing in a market that is saturated with other businesses selling the same product or service. It is increasingly vital to create brand differentiation right from the very beginning to stand-out in the market. By authentically expressing their brand identity and adopting a systematic brand-building approach, we help SMEs and start-ups build their own valuable brands for stronger sales and better business performance.

    ReplyDelete
  14. Great content it was really attractive


    Construction companies often request strong, masculine logos with bold colours. Whether you’re looking for something similar or to break from the norm, we make it easy to get a design that’s right for you. Brochures are perfect for promoting and providing essential information about the business. A clean, professional, and high-impact designed brochure sets you apart from your competition and reinforces the value your company offers. Ideal Design makes brochures Design different. Unique designs and strategic objectives turn prospective leads into clients or customers.


    http://idealbranding.in/

    idealbranding05@gmail.com

    ReplyDelete