Sending and Receiving JSON messages in Kafka

Sometime back i wrote couple of articles for Java World about Kafka Big data messaging with Kafka, Part 1 and Big data messaging with Kafka, Part 2, you can find basic Producer and Consumer for Kafka along with some basic samples.
I wanted to figure out how do i pass JSON message using Kafka. It looks like Kafak Connect provides a simple JSON Serializer org.apache.kafka.connect.json.JsonSerializer and Desrializer org.apache.kafka.connect.json.JsonDeserializer that uses Jackson JSON parser. I wanted to figure out how to use it, so i built following sample
  • First i did create a Contact Object, which is a simple Pojo that has 3 fields contactId, firstName and lastName. Take a look at main() method, in which i create simple object of Contact and then convert it to JSON and write to console.
  • Next i created Producer.java, which reads values in CSV format like 1,Sunil,Patil from command line and parse it to Contact object first. Then i convert Contact object into JSONNode and pass it as value to Kafka, The JSONSerializer converts the JsonNode into byte[]
    The producer code is mostly same as one required for passing String, with difference that on line 35, i am creating object of com.fasterxml.jackson.databind.ObjectMapper and then on line 41 i am converting Contact object into JSONNode by calling objectMapper.valueToTree(contact)
  • Since i am using org.apache.kafka.connect.json.JsonSerializer on the producer i have to use org.apache.kafka.connect.json.JsonDeserializer on the Consumer, Then while creating KafkaConsumer object i declare that i will get String key and JSONNode as value. Then once i get messages from Kafka i am calling mapper.treeToValue(jsonNode,Contact.class) to read the message and convert it back to Contact object.
Now you can run the producer and consumer with same topic name and it should work

109 comments:

  1. Thanks man very useful example

    ReplyDelete
    Replies
    1. Sometime back i wrote couple of articles for Java World about Kafka Big data messaging with Kafka, Part 1 and Big data messaging with Kafka, Part 2, you can find basic Producer and Consumer for Kafka along with some basic samples.
      I wanted to figure out how do i pass JSON message using Kafka. It looks like Kafak Connect provides a simple JSON Serializer org.apache.kafka.connect.json.JsonSerializer and Desrializer org.apache.kafka.connect.json.JsonDeserializer that uses Jackson JSON parser. I wanted to figure out how to use it, so i built following sample

      Deep Learning Projects for Final Year

      Machine Learning Final Year Projects

      Big Data Projects For Final Year Students

      Delete
  2. While consuming the messages in java, the json object is not getting to java console

    ReplyDelete
  3. ConsumerRecords records = kafkaConsumer.poll(100);

    here records size coming as Zero

    ReplyDelete
  4. Thanks man really helpful !!!

    ReplyDelete
  5. Team, can you please help on this issue?

    After closing KafkaConsumer
    Exception in thread "main" org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Hello': was expecting ('true', 'false' or 'null')
    at [Source: [B@4313f5bc; line: 1, column: 7]
    Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Hello': was expecting ('true', 'false' or 'null')
    at [Source: [B@4313f5bc; line: 1, column: 7]
    at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
    at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
    at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3466)

    ReplyDelete
  6. This was a live saver. Thank you!

    ReplyDelete
  7. Nice work!!! but this doesn't support for the nested classes within the JSON message. please give an example having nested classes in the JSON message.

    ReplyDelete

  8. Thanks for helping me to understand concepts. As a beginner in Hadoop your post help me a lot.
    Hadoop Training in Velachery | Hadoop Training

    ReplyDelete
  9. Existing without the answers to the difficulties you’ve sorted out through this guide is a critical case, as well as the kind which could have badly affected my entire career if I had not discovered your website. Best AWS Training in Bangalore

    ReplyDelete
  10. Hi There,

    Hot! That was HOT! Glued to the "Sending and Receiving JSON messages in Kafka" your proficiency and style!

    i'm trying to update a record using "onrowaction " button in lightning datatable , when i'm passing the parameters to server side controller getting this error "Cannot read property 'updaterecord' of undefined".

    Please help me to find solutions for below querrys
    1. How to capture row id and pass to server side controller .
    2. How to get the component value ( getting error 'Cannot read property 'updaterecord' of undefined')
    2. Is it possible to call helper method using "onrowaction " . Below my sample code

    Thank you very much and will look for more postings from you.

    Many Thanks,
    Preethi.

    ReplyDelete
  11. Hi There,

    Brilliant article, glad I slogged through the "Sending and Receiving JSON messages in Kafka" it seems that a whole lot of the details really come back to from my past project.

    I have a custom field on my Account object, say 'refered_by__c'. This field contains the ID of another Account object.

    I want to write a SOQL something like this (psedu query):

    select id, name, refered_by__c, from account

    Please advise how can I write this query using SOQL.

    By the way do you have any YouTube videos, would love to watch it. I would like to connect you on LinkedIn, great to have experts like you in my connection (In case, if you don’t have any issues).


    Kind Regards,
    Preethi.

    ReplyDelete
  12. Hi Your Blog Is Very Nice! Attractive. Content is Nice

    Stumagz is ultimate platform to release student magazine articles and engineering college news. It helps students to share their ideas in form of article.

    stuMagz is an online platform that brings all the students and colleges together. Despite the fact that NBA accreditation says that every college must maintain a magazine to publish their content, there is no proper platform for students to expose themselves to the ecosystem and the lack of reach for the students to the various opportunities present in the market.There is a lack of connectivity between different colleges and universities.

    The idea of stuMagz was born to bridge this gap and expand the scope for learning. It is a simple and efficient platform which provides every college and its students, a hassle free experience to publish their content and unleash their creativity.

    Student Magazine Articles

    Digital Campus Eco-System

    Digital Stories In Hyderabad

    Digital Classrooms In Hyderabad

    Student Magazine Subscriptions

    College Magazine Articles

    College Fest Event

    Top Engineering Colleges In India

    For more Details Visit Us: Stumagz.com

    ReplyDelete
  13. Really very nice blog information for this one and more technical skills are improved, I like that kind of post.
    Devops training in Chennai
    Devops training in Bangalore
    Devops Online training
    Devops training in Pune

    ReplyDelete
  14. I have read a few of the articles on your website now, and I really like your style of blogging. I added it to my favourites blog site list and will be checking back soon.

    java training in tambaram | java training in velachery

    java training in omr | oracle training in chennai

    java training in annanagar | java training in chennai

    ReplyDelete
  15. Thanks Admin for sharing such a useful post, I hope it’s useful to many individuals for developing their skill to get good career.
    Blueprism training in velachery

    Blueprism training in marathahalli

    ReplyDelete
  16. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.

    angularjs-Training in velachery

    angularjs-Training in annanagar

    angularjs Training in chennai

    angularjs Training in chennai

    ReplyDelete
  17. UiPath Training in Bangalore by myTectra is one the best UiPath Training. myTectra is the market leader in providing Robotic Process Automation on UiPath
    ui path training in bangalore

    ReplyDelete
  18. Thank you for this post. Thats all I are able to say. You most absolutely have built this blog website into something speciel. You clearly know what you are working on, youve insured so many corners.thanks
    angularjs-Training in pune

    angularjs-Training in chennai

    angularjs Training in chennai

    angularjs-Training in tambaram

    angularjs-Training in sholinganallur

    ReplyDelete
  19. I likable the posts and offbeat format you've got here! I’d wish many thanks for sharing your expertise and also the time it took to post!!
    python training in chennai
    python training in Bangalore
    Python training institute in chennai

    ReplyDelete
  20. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
    Devops training in sholinganallur
    Devops training in velachery
    Devops training in annanagar
    Devops training in tambaram

    ReplyDelete
  21. I am a regular reader of your blog and being students it is great to read that your responsibilities have not prevented you from continuing your study and other activities. Love

    angularjs Training in marathahalli

    angularjs interview questions and answers

    angularjs Training in bangalore

    angularjs Training in bangalore

    angularjs online Training

    angularjs Training in marathahalli

    ReplyDelete
  22. Thanks for such a great article here. I was searching for something like this for quite a long time and at last I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays.pmp training in chennai | pmp training institute in chennai | pmp training centers in chennai| pmp training in velachery | pmp training near me | pmp training courses online

    ReplyDelete
  23. Thanks for sharing such a nice info.I hope you will share more information like this. Please
    keep on sharing!

    mbaguide
    Article submission sites

    ReplyDelete
  24. your blog information's are really creative and It contains full of new innovative ideas.
    thank you for sharing with us.please update more data.
    Java Training
    Java training courses near me
    Java Training Institute in Vadapalani
    Java Courses in OMR

    ReplyDelete
  25. thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post.
    Devops Training in Chennai | Devops Training Institute in Chennai

    ReplyDelete
  26. You know your projects stand out of the herd. There is something special about them. It seems to me all of them are really brilliant!
    big data course malaysia

    ReplyDelete
  27. Learn how to get Norton security on your device. Also, find steps for installation and activation of Norton setup. Login to your Norton account at norton.com/setup.

    ReplyDelete
  28. Mobile texting service USA is helping its customers to expand their business with the best services of branded SMS in the USA. With this customers can send SMS for Campaigns, Discount, Alerts, and sales. Use SMS Marketing in the USA to get more customers and increase profit.

    ReplyDelete

  29. If you are thinking to create the latest Android apps, likes games, music, movies, TV, books, magazines & more then join us 4Square Logic IT Solutions we are providing you best Android app Development services within your budget. Our team of developers making the best attractive apps which is easy to use and enhance your business. For more query call us +91-7696224488

    ReplyDelete
  30. QuickBooks Com Error Crash affects your system when you try to email Invoices along with other transactions. This can be resolve by updating your windows.

    ReplyDelete
  31. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!
    rpa taining in malaysia

    ReplyDelete
  32. Today Telugu news updates provide us the information of breaking news and live updates. we get live news, political, education, technology, etc. Today Telugu news gives the best news updates. It also keeps its readers informed about the latest happenings in the world with instant updates.

    ReplyDelete
  33. It’s always so sweet and also full of a lot of fun for me personally and my office colleagues to search your blog a minimum of thrice in a week to see the new guidance you have got.

    Best PHP Training Institute in Chennai|PHP Course in chennai
    Best .Net Training Institute in Chennai
    Dotnet Training in Chennai
    Dotnet Training in Chennai

    ReplyDelete
  34. Really very nice blog information for this one and more technical skills are improve,i like that kind of post.Automation Anywhere Training in Bangalore

    ReplyDelete
  35. Thanks for sharing this information...
    AngularJS Training in Bangalore | AngularJS Course Fees | AngularJS 6 - i Digital Academy - AngularJS Training in Bangalore - Learn AngularJS 6 from Expert Real-time Trainers
    at i Digital Academy with Live Projects and Placement Assistance. Book a Free Demo Today.
    https://www.idigitalacademy.com/angularjs-training-in-bangalore/

    ReplyDelete
  36. I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post. Hats off to you! The information that you have provided is very helpful.
    data science course
    data science interview questions

    ReplyDelete
  37. Thanks for sharing this article...
    Best AWS with Devops Training in Bangalore | AWS with Devops Training Course Content | AWS with Devops Training Institutes | AWS with Devops Online Training - Elegant IT Services
    - Elegant IT Services provides Best AWS with Devops Training in Bangalore with expert real-time trainers who are working Professionals with min 8 + years of experience in AWS with Devops Training Industry, we also provide 100% Placement Assistance with Live Projects on AWS with Devops Training.

    ReplyDelete
  38. Nice blog,I understood the topic very clearly,And want to study more like this.
    CCC Question Paper

    ReplyDelete
  39. It’s always so sweet and also full of a lot of fun for me personally and my office colleagues to search your blog a minimum of thrice in a week to see the new guidance you have got.

    7 tips to start a career in digital marketing

    “Digital marketing is the marketing of product or service using digital technologies, mainly on the Internet, but also including mobile phones, display advertising, and any other digital medium”. This is the definition that you would get when you search for the term “Digital marketing” in google. Let’s give out a simpler explanation by saying, “the form of marketing, using the internet and technologies like phones, computer etc”.

    we have offered to the advanced syllabus course digital marketing for available join now.

    more details click the link now

    https://www.webdschool.com/digital-marketing-course-in-chennai.html

    ReplyDelete
  40. Thank for this blog are more informative step by step and useful contents. I here by attached my site would you see this blog

    Web designing trends in 2020

    When we look into the trends, everything which is ruling today’s world was once a start up and slowly begun getting into. But Now they have literally transformed our lives on a tremendous note. To name a few, Facebook, Whats App, Twitter can be a promising proof for such a transformation and have a true impact on the digital world.

    we have offered to the advanced syllabus course web design and development for available join now

    more details click the link now

    https://www.webdschool.com/web-development-course-in-chennai.html

    ReplyDelete
  41. Study Business Analytics Course in Bangalore with ExcelR where you get a great experience and better knowledge.
    Business Analytics Course

    ReplyDelete
  42. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained!
    data science certification

    ReplyDelete
  43. I think about it is most required for making more on this get engaged
    data scientist course in malaysia

    ReplyDelete
  44. This comment has been removed by the author.

    ReplyDelete
  45. This comment has been removed by the author.

    ReplyDelete
  46. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science online course

    ReplyDelete
  47. Excellent post. I was always checking this blog, and I’m impressed! Extremely useful info specially the last part, I care for such information a lot. I was exploring this particular info for a long time. Thanks to this blog my exploration has ended.
    If you want Digital Marketing Serives :-
    Digital marketing Service in Delhi
    SMM Services
    PPC Services in Delhi
    Website Design & Development Packages
    SEO Services PackagesLocal SEO services
    E-mail marketing services
    YouTube plans

    ReplyDelete
  48. Great blog informative for readers such an explicit content stay posting thanks for sharing.
    We Are Providing Excellence seo services chicago

    ReplyDelete
  49. A Truly Stunning Vintage WW2 B3 sheepskin shearling flying pilot leather bomber jacket size Large.

    This Garment Is In Very Good Condition For It's Age With No Rips, Holes Or Tears & The Internal Sheepskin Lining Is Also In Excellent Condition.

    There Does Appear To Be Some Signs Of General Wear, Repairs, Scuffs, Nicks Etc, Etc Etc In Places Which Only Further Enhances The Natural Character & Patina Of This Item And Is Due To The Unique History Of The Garment.

    It Also Appears to Have Had Some Restoration Work, New Zip Etc At Some Point In Order To Restore, And Preserve This Wonderful Garment.

    ReplyDelete
  50. thanks for sharing information awesome blog post
    Creative World

    ReplyDelete
  51. Awesome Article Love to read a full article. SEO Consultants Chicago

    ReplyDelete
  52. Mready mens leather bomber jackets have outstanding combination of premium quality real leather and top quality extra fine elastic rib. Very soft quilt lining used inside will feel you more relaxed in both indoor and outdoor activities.

    ReplyDelete
  53. Hey ,

    Great Job . You Know what ?

    I read a lot of blog posts and I never heard of such a topic. I love the subject you have done about bloggers. Very simple. I am also writing a blog related to the Germany job seeking visa. You can also see this.

    ReplyDelete
  54. Nice talk, I would like to add here that I’m a Software Developer Outsource living in Chicago. I am a fan of Outsourcing, Software Development, and Web Application Developer. I’m also interested in DevOps Outsourcing. You can hire me with a click on the button above. If you want more details to contact us through our website page.

    ReplyDelete
  55. I was very pleased to find this site.I wanted to thank you for this great read!! I definitely enjoy every little bit of it and I have you bookmarked to check out new stuff you post.

    business analytics course

    ReplyDelete
  56. I was ecstatic to discover this website. I wanted to express my gratitude for this fantastic read!!

    ReplyDelete
  57. YouTube is one of the largest social media platform all over the world. It is a free video sharing website that makes it easy to watch online videos. It is the second largest search engine after Google. And you’ll find all kinds of youtubers here-Cooking, Entertainment, Comedy, Tech etc. due to their interest. Like wise I posted this article of the top 10 youtubers income in Hyderabad who have more subscribers, views, videos, income from top 1 to top 10.

    ReplyDelete
  58. Very much informative, thanks for sharing.
    SEO Training In Hyderabad

    ReplyDelete
  59. QuickBooks Tool Hub is a tool that is a collection of other sub tools like QuickBooks File Doctor and QuickBooks Database Server Manager tool. So you must download this tool as it has the capability to deal with most of the errors.

    ReplyDelete
  60. It's been our pleasure to inform you that our institution is offering a great deal by giving CS executive classes and a free CSEET class only for those who are interested in gaining knowledge. So what are you waiting for contact us or visit our website at https://uniqueacademyforcommerce.com/

    ReplyDelete
  61. Excellent blog, I wish to share your post with my folks circle. It’s really helped me a lot, so keep sharing post like this…

    Data Science Training in Hyderabad

    ReplyDelete
  62. 더킹카지노,퍼스트카지노,더나인카지노,예스카지노,샌즈카지노,007카지노,코인카지노,더존카지노를 추천하는 CASINO-KOREA는 100% 검증을 거친 가장 안전한 카지노.온라인카지노

    ReplyDelete
  63. I have read your excellent post. This is a great job. I have enjoyed reading your post first time. I want to say thanks for this post. Thank you...
    data scientist certification malaysia

    ReplyDelete
  64. Grab the Digital Marketing Training in Chennai from Infycle Technologies, the best software training institute, and Placement center in Chennai which is providing professional software courses such as Data Science, Artificial Intelligence, Cyber Security, Big Data, Java, Hadoop, Selenium, Android, and iOS Development, DevOps, Oracle etc with 100% hands-on practical training. Dial 7502633633 to get more info and a free demo and to grab the certification for having a peak rise in your career.

    ReplyDelete

  65. Thanks for giving this beautiful post. It is great article; even this is a very important topic. job news bd I find your site very useful. Many thanks for sharing these lovely tips. Many things can be done with this software. how to send pictures on tinder Specialty it is very helpful for beginner. I also learned a lot from your site. Thank you very much for sharing your post with us.https://nokiasoftware.net/
    best clipping path service provider
    https://clippingpathgenius.com/

    ReplyDelete
  66. This is the first time I visit here. I found such a large number of engaging stuff in your blog, particularly its conversation. From the huge amounts of remarks on your articles, I surmise I am by all accounts not the only one having all the recreation here! Keep doing awesome. I have been important to compose something like this on my site and you have given me a thought.

    ReplyDelete
  67. 360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

    ReplyDelete
  68. data analytics courseApril 18, 2022 at 10:42 PM

    Great to become visiting your weblog once more, it has been a very long time for me. Pleasantly this article i've been sat tight for such a long time. I will require this post to add up to my task in the school, and it has identical subject along with your review. Much appreciated, great offer.data analytics course

    ReplyDelete
  69. People are impressed with this technology, and the experts have predicted a bright future of data science.

    ReplyDelete

  70. Wohh I love it, I mean this blog is just awesome. I personally love this informative article… Keep Posting such information… Keep Going!
    USA Dedicated Server

    ReplyDelete
  71. Hey, your blog ideas are very amazing and informative. If you are looking for the fastest and cheapest Canada VPS Server

    ReplyDelete

  72. Wow, I like it, this blog is just amazing. Personally love this informative article… Keep Posting such information. Finland VPS Server

    ReplyDelete
  73. Thanks for sharing such an amazing content. Six Phrase stands as India’s foremost Skill Development and Career Advancement Enterprise. Enhance your skills with our comprehensive Big Data Training. Join us today and become an expert in Six Phrase techniques.

    ReplyDelete