Sending and Receiving JSON messages in Kafka

Sometime back i wrote couple of articles for Java World about Kafka Big data messaging with Kafka, Part 1 and Big data messaging with Kafka, Part 2, you can find basic Producer and Consumer for Kafka along with some basic samples.
I wanted to figure out how do i pass JSON message using Kafka. It looks like Kafak Connect provides a simple JSON Serializer org.apache.kafka.connect.json.JsonSerializer and Desrializer org.apache.kafka.connect.json.JsonDeserializer that uses Jackson JSON parser. I wanted to figure out how to use it, so i built following sample
  • First i did create a Contact Object, which is a simple Pojo that has 3 fields contactId, firstName and lastName. Take a look at main() method, in which i create simple object of Contact and then convert it to JSON and write to console.
  • Next i created Producer.java, which reads values in CSV format like 1,Sunil,Patil from command line and parse it to Contact object first. Then i convert Contact object into JSONNode and pass it as value to Kafka, The JSONSerializer converts the JsonNode into byte[]
    The producer code is mostly same as one required for passing String, with difference that on line 35, i am creating object of com.fasterxml.jackson.databind.ObjectMapper and then on line 41 i am converting Contact object into JSONNode by calling objectMapper.valueToTree(contact)
  • Since i am using org.apache.kafka.connect.json.JsonSerializer on the producer i have to use org.apache.kafka.connect.json.JsonDeserializer on the Consumer, Then while creating KafkaConsumer object i declare that i will get String key and JSONNode as value. Then once i get messages from Kafka i am calling mapper.treeToValue(jsonNode,Contact.class) to read the message and convert it back to Contact object.
Now you can run the producer and consumer with same topic name and it should work

81 comments:

farhoud mojahedzadeh said...

Thanks man very useful example

venkatreddy katla said...

While consuming the messages in java, the json object is not getting to java console

venkatreddy katla said...

ConsumerRecords records = kafkaConsumer.poll(100);

here records size coming as Zero

venkatreddy katla said...

Thanks It is working....

Anonymous said...

Thanks man really helpful !!!

Anonymous said...

Team, can you please help on this issue?

After closing KafkaConsumer
Exception in thread "main" org.apache.kafka.common.errors.SerializationException: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Hello': was expecting ('true', 'false' or 'null')
at [Source: [B@4313f5bc; line: 1, column: 7]
Caused by: com.fasterxml.jackson.core.JsonParseException: Unrecognized token 'Hello': was expecting ('true', 'false' or 'null')
at [Source: [B@4313f5bc; line: 1, column: 7]
at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1586)
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:521)
at com.fasterxml.jackson.core.json.UTF8StreamJsonParser._reportInvalidToken(UTF8StreamJsonParser.java:3466)

Anonymous said...

This was a live saver. Thank you!

Mounika Karoshi said...

Good
python training in bangalore
python online training

KATE TECHNOLOGIES said...

hi your blog Article is very nice & thanks for sharing the information.
seo services in warangal
web design services in warangal
android services in warangal

Amar G said...

Thank you for sharing this type of interview questions
Iot Online Training
Itil Interview Questions
Salesforce Interview Questions
Msbi Interview questions
Salesforce Interview Questions
C Interview Questions

Anonymous said...

Nice work!!! but this doesn't support for the nested classes within the JSON message. please give an example having nested classes in the JSON message.

regina razz said...



with your entire process to setup office product online. Have you Just bought Microsoft Office product ? If yes then you can complete your Office Setup online with your product key code. You just need open office.com/setup , Install Office , Install Microsoft Office or setup.office.com into your web browser.

office setup
www.office.com/setup
office.com/setup

regina razz said...


In case any message saying ‘Stop’ pops up while installing Microsoft Office 365 due to a compatibility issue, contact our live chat support.Our online Experts through live chat will guide you through the entire process of Office setup, covering all steps and issues.Keep your 25 characters long product key with you.



Office Com Setup
office com/setup
www.office.com/setup

sowmiya gopal said...


Thanks for helping me to understand concepts. As a beginner in Hadoop your post help me a lot.
Hadoop Training in Velachery | Hadoop Training

gkr ragini said...

Existing without the answers to the difficulties you’ve sorted out through this guide is a critical case, as well as the kind which could have badly affected my entire career if I had not discovered your website. Best AWS Training in Bangalore

preethi Sharma said...

Hi There,

Hot! That was HOT! Glued to the "Sending and Receiving JSON messages in Kafka" your proficiency and style!

i'm trying to update a record using "onrowaction " button in lightning datatable , when i'm passing the parameters to server side controller getting this error "Cannot read property 'updaterecord' of undefined".

Please help me to find solutions for below querrys
1. How to capture row id and pass to server side controller .
2. How to get the component value ( getting error 'Cannot read property 'updaterecord' of undefined')
2. Is it possible to call helper method using "onrowaction " . Below my sample code

Thank you very much and will look for more postings from you.

Many Thanks,
Preethi.

preethi Sharma said...

Hi There,

Brilliant article, glad I slogged through the "Sending and Receiving JSON messages in Kafka" it seems that a whole lot of the details really come back to from my past project.

I have a custom field on my Account object, say 'refered_by__c'. This field contains the ID of another Account object.

I want to write a SOQL something like this (psedu query):

select id, name, refered_by__c, from account

Please advise how can I write this query using SOQL.

By the way do you have any YouTube videos, would love to watch it. I would like to connect you on LinkedIn, great to have experts like you in my connection (In case, if you don’t have any issues).


Kind Regards,
Preethi.

Stumagz.com said...

Hi Your Blog Is Very Nice! Attractive. Content is Nice

Stumagz is ultimate platform to release student magazine articles and engineering college news. It helps students to share their ideas in form of article.

stuMagz is an online platform that brings all the students and colleges together. Despite the fact that NBA accreditation says that every college must maintain a magazine to publish their content, there is no proper platform for students to expose themselves to the ecosystem and the lack of reach for the students to the various opportunities present in the market.There is a lack of connectivity between different colleges and universities.

The idea of stuMagz was born to bridge this gap and expand the scope for learning. It is a simple and efficient platform which provides every college and its students, a hassle free experience to publish their content and unleash their creativity.

Student Magazine Articles

Digital Campus Eco-System

Digital Stories In Hyderabad

Digital Classrooms In Hyderabad

Student Magazine Subscriptions

College Magazine Articles

College Fest Event

Top Engineering Colleges In India

For more Details Visit Us: Stumagz.com

srjwebsolutions said...


We are leading responsive website designing and development company in Noida.
We are offering mobile friendly responsive website designing, website development, e-commerce website, seo service and sem services in Noida.

Responsive Website Designing Company in Noida
Website Designing Company in Noida
SEO Services in Noida
SMO Services in Noida

Vikas Chaudhary said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

EG MEDI said...

Egmedi.com is online medical store pharmacy in laxmi nagar Delhi. You can Order prescription/OTC medicines online. Cash on Delivery available. Free Home Delivery


Online Pharmacy in Delhi
Buy Online medicine in Delhi
Online Pharmacy in laxmi nagar
Buy Online medicine in laxmi nagar
Onine Medical Store in Delhi
Online Medical store in laxmi nagar
Online medicine store in delhi
online medicine store in laxmi nagar
Purchase Medicine Online
Online Pharmacy India
Online Medical Store

viswanath said...

AWS Training in Bangalore - Live Online & Classroom
myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

IOT Training in Bangalore - Live Online & Classroom
IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

shalinipriya said...

Inspiring writings and I greatly admired what you have to say , I hope you continue to provide new ideas for us all and greetings success always for you..Keep update more information..
Data Science Training in Chennai
Data science training in bangalore
Data science online training
Data science training in pune
Data science training in kalyan nagar

Saro said...

Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

Saro said...

Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

Saro said...

It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command

automation anywhere training in chennai

automation anywhere training in bangalore

automation anywhere training in pune

automation anywhere online training

blueprism online training

rpa Training in sholinganallur

rpa Training in annanagar

iot-training-in-chennai

blueprism-training-in-pune

automation-anywhere-training-in-pune

Saro said...

Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

Saro said...

It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command

automation anywhere training in chennai

automation anywhere training in bangalore

automation anywhere training in pune

automation anywhere online training

blueprism online training

rpa Training in sholinganallur

rpa Training in annanagar

iot-training-in-chennai

blueprism-training-in-pune

automation-anywhere-training-in-pune

Saro said...

This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

rpa Training in tambaram

blueprism Training in tambaram

automation anywhere training in tambaram

iot Training in tambaram

rpa training in sholinganallur

blue prism training in sholinganallur

automation anywhere training in sholinganallur

iot training in sholinganallur


Saro said...

Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

nivatha said...

Really very nice blog information for this one and more technical skills are improved, I like that kind of post.
Devops training in Chennai
Devops training in Bangalore
Devops Online training
Devops training in Pune

digi mark said...

Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

digi mark said...

It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command

automation anywhere training in chennai

automation anywhere training in bangalore

automation anywhere training in pune

automation anywhere online training

blueprism online training

rpa Training in sholinganallur

rpa Training in annanagar

iot-training-in-chennai

blueprism-training-in-pune

automation-anywhere-training-in-pune

simbu said...

Outstanding blog post, I have marked your site so ideally I’ll see much more on this subject in the foreseeable future.
java training in annanagar | java training in chennai

java training in marathahalli | java training in btm layout

java training in rajaji nagar | java training in jayanagar

java training in chennai

shalinipriya said...

All the points you described so beautiful. Every time i read your i blog and i am so surprised that how you can write so well.
Data Science training in marathahalli
Data Science training in btm
Data Science training in rajaji nagar
Data Science training in chennai
Data Science training in kalyan nagar
Data Science training in electronic city
Data Science training in USA
Data science training in pune

simbu said...

I have read a few of the articles on your website now, and I really like your style of blogging. I added it to my favourites blog site list and will be checking back soon.

java training in tambaram | java training in velachery

java training in omr | oracle training in chennai

java training in annanagar | java training in chennai

ganga said...

Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.

angularjs Training in bangalore

angularjs Training in btm

angularjs Training in electronic-city

angularjs Training in online

angularjs Training in marathahalli

pragya pragya said...

It would have been the happiest moment for you,I mean if we have been waiting for something to happen and when it happens we forgot all hardwork and wait for getting that happened.
python training in pune
python online training
python training in OMR

ummayasri said...

Thanks Admin for sharing such a useful post, I hope it’s useful to many individuals for developing their skill to get good career.
Blueprism training in velachery

Blueprism training in marathahalli

sweety ganga said...

This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.

angularjs-Training in velachery

angularjs-Training in annanagar

angularjs Training in chennai

angularjs Training in chennai

Anonymous said...

UiPath Training in Bangalore by myTectra is one the best UiPath Training. myTectra is the market leader in providing Robotic Process Automation on UiPath
ui path training in bangalore

Rohil singh said...

Great post and informative blog.it was awesome to read, thanks for sharing this great content to my vision.
Good discussion.
PMP Training Course in Bangalore
PMP Training Course in Dammam
PMP Training Course in Dubai
PMP Training Course in Jeddah
PMP Training Course in Riyadh

sathya shri said...

Thank you for this post. Thats all I are able to say. You most absolutely have built this blog website into something speciel. You clearly know what you are working on, youve insured so many corners.thanks
angularjs-Training in pune

angularjs-Training in chennai

angularjs Training in chennai

angularjs-Training in tambaram

angularjs-Training in sholinganallur

Unknown said...

myTectra Placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance

Rohil singh said...

Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us. Do check Six Sigma Training in Bangalore | Six Sigma Training in Dubai & Get trained by an expert who will enrich you with the latest trends.

pragya pragya said...

I likable the posts and offbeat format you've got here! I’d wish many thanks for sharing your expertise and also the time it took to post!!
python training in chennai
python training in Bangalore
Python training institute in chennai

gowsalya said...

Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
Devops training in sholinganallur
Devops training in velachery
Devops training in annanagar
Devops training in tambaram

amala jst said...

A very nice guide. I will definitely follow these tips. Thank you for sharing such detailed article. I am learning a lot from you.

rpa training in electronic-city | rpa training in btm | rpa training in marathahalli | rpa training in pune

Anonymous said...

I am a regular reader of your blog and being students it is great to read that your responsibilities have not prevented you from continuing your study and other activities. Love

angularjs Training in marathahalli

angularjs interview questions and answers

angularjs Training in bangalore

angularjs Training in bangalore

angularjs online Training

angularjs Training in marathahalli

Sadhana Rathore said...

Useful content, I have bookmarked this page for my future reference.
DevOps Training in Chennai
DevOps course in Chennai
DevOps Training
Robotics Process Automation Training in Chennai
Angularjs Training in Chennai
AWS Certification in Chennai

Anbarasan14 said...

Thanks for this kind of worthy information. this was really very helpful to me. keep continuing.

French Language Course in Chennai
French Training in Chennai
Spanish Language Classes in Chennai
Spanish Classes in Chennai
Japanese Language Training in Chennai
Japanese Classes near me
Japanese Classes in Chennai

Shiva Shakthi said...

Really the blog which you have shared is more informative to us. Thanks for your information.
Tally Classes in Coimbatore
Tally Course in Coimbatore
Tally Training Institute
Best Tally Course
Tally Training Coimbatore

Gautam krish said...

Thanks for your blog. The information which you have shared is excellent.
Oracle Course
Oracle dba Course
Oracle SQL Certification
Oracle Certification Course
Oracle Database Course

Robotic Process Automation Tutorial said...

Thanks for such a great article here. I was searching for something like this for quite a long time and at last I’ve found it on your blog. It was definitely interesting for me to read about their market situation nowadays.pmp training in chennai | pmp training institute in chennai | pmp training centers in chennai| pmp training in velachery | pmp training near me | pmp training courses online

yuva rani said...

Excellent Blog!!! Such an interesting blog with clear vision, this will definitely help for beginner to make them update.
Data Science Training in Bangalore
Data Science Certification Bangalore
Data Science in Bangalore
Devops Training in Bangalore
Devops Training Institutes in Bangalore
Devops Certification in Bangalore

Vicky Ram said...

Thanks for sharing such a nice info.I hope you will share more information like this. Please
keep on sharing!

mbaguide
Article submission sites

mercyroy said...

your blog information's are really creative and It contains full of new innovative ideas.
thank you for sharing with us.please update more data.
Java Training
Java training courses near me
Java Training Institute in Vadapalani
Java Courses in OMR

Nadeem Malik said...

hello sir,
thanks for giving that type of information.
digital marketing company in delhi
HP DesignJet T520 In Delhi

LindaJasmine said...


I love your way of writing. The content shows your in-depth knowledge on the subject. Thanks for Sharing.
Node JS Training in Chennai
Node JS Course in Chennai
Node JS Advanced Training
Node JS Training Institute in chennai
Node JS Course
IELTS coaching in Chennai
IELTS Training in Chennai
SAS Training in Chennai
SAS Course in Chennai

Mr Solution said...

How To Make Money From Youtube
Web Development
How To Approve Google Adsense Account
Seo Marketing
How To Make Free Website
Ecommerce Education
Mobile Software Course English/Urdu/Hindi

Akash Thakur said...

Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.
interior designer company in noida

Amase LED Lighting said...

Home Decor Wall Lights in delhi
Thanks for giving great kind of information. So useful and practical for me. Thanks for your excellent blog, nice work keep it up thanks for sharing the knowledge.

LindaJasmine said...


Extra-ordinary post. Looking for an information like this for a long time. Thanks for Posting.
Pega training in chennai
Pega course in chennai
Primavera Training in Chennai
Primavera Course in Chennai
IELTS Coaching in Chennai
IELTS Training in Chennai

peter berg said...

MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.

www.office.com/setup

jaya devan said...

Very Clear Explanation. Thank you to share this
Regards,
Best Devops Training in Chennai | Best Devops Training Institute in Chennai

Lucky Patcher said...

whatsapp group links

Riya Raj said...

Wonderful Blog!!! Thanks for sharing with us...
German Classes in Bangalore
German Language Classes in Bangalore
German Classes in Coimbatore
German language classes in Coimbatore
German Classes in Madurai
PHP Course in Madurai
Spoken English Class in Madurai
Selenium Training in Coimbatore
SEO Training in Coimbatore
Web Designing Course in Madurai

suresh said...

Very impressive thanks for sharing
DevOps Training in Chennai

Cloud Computing Training in Chennai

IT Software Training in Chennai


Hema Sri said...


Thank you for giving the information and it is useful for me. training with placement company in Hyderabad

Kavi said...

thank you for this great read!! I definitely enjoying every little bit of it I have you bookmarked to check out new stuff you post.
Devops Training in Chennai | Devops Training Institute in Chennai

Logo Design said...

Thanks for sharing useful information - visit - Logo Design Company in Jaipur

Sadhana Rathore said...

Your site is amazing and your blogs are informative. I have bookmarked this blog for my reference.
Automation Anywhere Training in Chennai
Automation Training Institute in Chennai
Automation Anywhere course in Chennai
Blue Prism Training Institute in Chennai
Blue Prism Training in Chennai
RPA course in Chennai

raybon said...

Thanks a lot for sharing us about this update. Hope you will not get tired on making posts as informative as this. 

devops online training

aws online training

data science with python online training

data science online training

rpa online training

rose said...

Have you been thinking about the power sources and the tiles whom use blocks I wanted to thank you for this great read!! I definitely enjoyed every little bit of it and I have you bookmarked to check out the new stuff you post
Microsoft Azure online training
Selenium online training
Java online training
Python online training
uipath online training

IELTS Coaching in Dwarka said...

I feel happy about and learning more about this topic. keep sharing your information regularly for my future reference. This content creates new hope and inspiration within me. Thanks for sharing an article like this. the information which you have provided is better than another blog.
IELTS Coaching in Dwarka

Vicky Ram said...

Nice post. I learned some new information. Thanks for sharing.

Guest posting sites
Technology

kamal said...

And indeed, I’m just always astounded concerning the remarkable things served by you. Some four facts on this page are undeniably the most effective I’ve had.
SOFTWARE TRAINING IN CHENNAI
POWERBI TRAINING IN CHENNAI
CCNA TRAINING IN CHENNAI
ANDROID TRAINING IN CHENNAI

Eiliza45 said...

I love this post.

โปรโมชั่นGclub ของทางทีมงานตอนนี้แจกฟรีโบนัส 50%
เพียงแค่คุณสมัคร Gclub กับทางทีมงานของเราเพียงเท่านั้น
ร่วมมาเป็นส่วนหนึ่งกับเว็บไซต์คาสิโนออนไลน์ของเราได้เลยค่ะ
สมัครสล็อตออนไลน์ >>> goldenslot
สนใจร่วมลงทุนกับเรา สมัครเอเย่น Gclub คลิ๊กได้เลย

Sathya said...

Interesting information and attractive.This blog is really rocking... Yes, the post is very interesting and I really like it.I never seen articles like this. I meant it's so knowledgeable, informative, and good looking site. I appreciate your hard work. Good job.
Kindly visit us @
Sathya Online Shopping
Online AC Price | Air Conditioner Online | AC Offers Online | AC Online Shopping
Inverter AC | Best Inverter AC | Inverter Split AC
Buy Split AC Online | Best Split AC | Split AC Online
LED TV Sale | Buy LED TV Online | Smart LED TV | LED TV Price
Laptop Price | Laptops for Sale | Buy Laptop | Buy Laptop Online
Full HD TV Price | LED HD TV Price
Buy Ultra HD TV | Buy Ultra HD TV Online
Buy Mobile Online | Buy Smartphone Online in India

lggy1688 said...

Very cool!

เว็บไซต์คาสิโนออนไลน์ที่ได้คุณภาพอับดับ 1 ของประเทศ
เป็นเว็บไซต์การพนันออนไลน์ที่มีคนมา สมัคร Gclub Royal1688
และยังมีหวยให้คุณได้เล่น สมัครหวยออนไลน์ ได้เลย
สมัครสมาชิกที่นี่ >>> Gclub Royal1688
ร่วมลงทุนสมัครเอเย่นคาสิโนกับทีมงานของเราได้เลย

kavithasathish said...

Thanks for this innovative blog. Keep posting the updates.
pearson vue
German Language Classes in Chennai
IELTS Training in Chennai
Japanese Language Course in Chennai
spanish classes in chennai
Best Spoken English Classes in Chennai
Spoken English Classes in Tambaram
Spoken English Classes in Anna Nagar