How to drain/delete/expire existing messages in Kafka

Some time you might have a bad record in Kafka topic that you want to delete. Kafka does not provide direct option to delete specific record. Only way to delete records is to expire them. You can achieve this by setting data retention to say 1 second that expires all the old messages. You can follow these steps
    First check the topic to find out value of retention.ms config parameter for the topic
  1. 
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name my-topic --entity-type topics
    Configs for topic 'my-topic' are retention.ms=86400000
    
  2. Change value of retention.ms to 1 which means all messages older than 1 ms will be expired
    
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name my-topic --entity-type topics --add-config retention.ms=1
    Completed Updating config for entity: topic 'my-topic'
    
  3. Wait for few seconds and monitor logs of the Kafka server to make sure that messages from the topic are deleted
  4. Now change the value of retention.ms back to its original value which was 86400000 (7 days)
    
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name my-topic --entity-type topics --add-config retention.ms=86400000
    Completed Updating config for entity: topic 'my-topic'.
    
  5. You can verify that your changes are saved by running this command
    
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name my-topic --entity-type topics
    Configs for topic 'my-topic' are retention.ms=86400000
    

36 comments:

Chaithra L said...

Its a nice post.
Best Oracle Training in Bangalore


ark santa hat said...

Super Einstieg! Das Offset-Verhalten beim Reset lässt sich auch über die Property `ConsumerConfig.AUTO_OFFSET_RESET_CONFIG` steuern.

ark bug repellent id said...

Hi! I got a video message on my instagram dm and I didn't know about the disappearing rule when you replay for couple of times. Is there any way or any hack that I can get the expired video back and able to save it? Its kinda important to me. Thank you.



eu4 console commands said...

Super Einstieg! Das Offset-Verhalten beim Reset lässt sich auch über die Property `ConsumerConfig.AUTO_OFFSET_RESET_CONFIG` steuern.


xcom2 console commands said...

why processing logic can't just put failed messages back into the topic if this is a business need? why to overcomplicate with additional component(redelivery tracker), additional topic etc. Seems like over-engineering.


ajay teja said...

Good work…unique site and interesting too… keep it up…looking forward for more updates.Good luck to all of you and thanks so much for your hard-work.
TranslationAndLocalizationCompanies,Transcription Services Bangalore,Closed Captioning Services in Bangalore

Shea butter soap benefits said...

I would like to know the operational cost relative to the use-case in traditional architecture as well as the mean time to recovery after a failure. Sometimes drawing a box around a complex system and giving it a fancy name is just an illusion or obfuscates the underlying complexity. While these systems seem simple in normal operation operation costs and recovery times are non-trivial.

Gokul Ravi said...

nice blog
android training in bangalore
ios training in bangalore

Gokul Ravi said...

useful blog
python interview questions
cognos interview questions
perl interview questions
vlsi interview questions
web api interview questions
msbi interview questions

Gokul Ravi said...

laravel interview questions
aem interview questions
salesforce interview questions
oops abab interview questions
itil interview questions
informatica interview questions
extjs interview questions

Gokul Ravi said...

sap bi interview questions
hive interview questions
seo interview questions
as400 interview questions
wordpress interview questions
accounting interview questions
basic accounting and financial interview questions

devidnayana said...

I am very happy when read this blog post because blog post written in good manner and write on good topic.
Thanks for sharing valuable information.
Web Design Company Bangalore,
Digital Marketing Company

Discover Amazing India- Ranjeet Kumar said...

Planning For A Memorable Trip To India! The Best And Most Preferred Vacation Package To Explore The Beauty Of India At Fullest Is With Enthralling Delhi Agra Jaipur Tour

Golden Triangle Tours
Same Day Tours
Rajasthan Tours

Discover Amazing India
+91-8266881111
info@discoveramazingindia.com
http://www.discoveramazingindia.com

srjwebsolutions said...

We are leading responsive website designing and development company in Noida. We are offering mobile friendly responsive website designing, website development, e-commerce website, seo service and sem services in Noida.

Responsive Website Designing Company in Noida
Website Designing Company in Noida
SEO Services in Noida
SMO Services in Noida

srjwebsolutions said...

We are leading responsive website designing and development company in Noida.
We are offering mobile friendly responsive website designing, website development, e-commerce website, seo service and sem services in Noida.

Responsive Website Designing Company in Noida
Website Designing Company in Noida
SEO Services in Noida
SMO Services in Noida

Vikas Chaudhary said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

Vikas Chaudhary said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

EG MEDI said...

Egmedi.com is online medical store pharmacy in laxmi nagar Delhi. You can Order prescription/OTC medicines online. Cash on Delivery available. Free Home Delivery


Online Pharmacy in Delhi
Buy Online medicine in Delhi
Online Pharmacy in laxmi nagar
Buy Online medicine in laxmi nagar
Onine Medical Store in Delhi
Online Medical store in laxmi nagar
Online medicine store in delhi
online medicine store in laxmi nagar
Purchase Medicine Online
Online Pharmacy India
Online Medical Store

Pandit Bhavesh Shastri said...

You’re All Problem Solution Shastri Ji. Famous Astrologer Love Problem Solution Expert And One Call Change Your Life. 100% Result In 24 Hours. Call & Whatsapp Chat On +91 8440824329

Online Love Vashikaran
Vashikaran Remove Astro
True Vashikaran Specialist
Online Vashikaran Specialist
Love Problem Solution
Husband Wife Problem Solution
Girl Vashikaran Mantra
Love Marriage Problem Solution
Get Lost Love Back Problem Solution
One Side Love Problem Solution


Nectar Phyto said...

great article. keep sharing
anybloginfo
anybloginfo
anybloginfo
anybloginfo
anybloginfo
anyblognews

viswanath said...

AWS Training in Bangalore - Live Online & Classroom
myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

IOT Training in Bangalore - Live Online & Classroom
IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

R.Alamsha Karnan said...

Hi Sunil

I like your Kafka blogs. BTW, i initiated a project to build a Kafka client for Ignition SCADA, based upon your blog at:
https://www.javaworld.com/article/3060078/big-data/big-data-messaging-with-kafka-part-1.html?page=2

Ignition forum:
https://forum.inductiveautomation.com/t/ignition-apache-kafka-build-massively-scalable-projects-demo/19317


Thanks for the nice article and neat code which works.

R.Alamsha

Saro said...

It's interesting that many of the bloggers to helped clarify a few things for me as well as giving.Most of ideas can be nice content.The people to give them a good shake to get your point and across the command

automation anywhere training in chennai

automation anywhere training in bangalore

automation anywhere training in pune

automation anywhere online training

blueprism online training

rpa Training in sholinganallur

rpa Training in annanagar

iot-training-in-chennai

blueprism-training-in-pune

automation-anywhere-training-in-pune

digi mark said...

Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.

rpa Training in Chennai

rpa Training in bangalore

rpa Training in pune

blueprism Training in Chennai

blueprism Training in bangalore

blueprism Training in pune

rpa online training

shalinipriya said...

Excellent blog, I wish to share your post with my folks circle. It’s really helped me a lot, so keep sharing post like this
Data Science Training in Chennai
Data science training in bangalore
Data science online training
Data science training in pune
Data science training in kalyan nagar
selenium training in chennai

simbu said...

This is such a great post, and was thinking much the same myself. Another great update.
java training in marathahalli | java training in btm layout

java training in jayanagar | java training in electronic city

java training in chennai | java training in USA

selenium training in chennai

Mouni yoga said...

Thank you for taking the time and sharing this information with us. It was indeed very helpful and insightful while being straight forward and to the point.
python training in pune
python online training
python training in OMR

shethal said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
Devops training in sholinganallur
Devops training in tambaram
DevOps online Training

ummayasri said...

Thanks a lot for sharing us about this update. Hope you will not get tired on making posts as informative as this. 
Blueprism training in Pune

Blueprism training in Chennai

Rohini .R said...

Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
Selenium Training in Bangalore | Selenium Training in Bangalore | Selenium Training in Bangalore | Selenium Training in Bangalore

chitra pragya said...

Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.
angularjs Training in chennai
angularjs-Training in pune

angularjs-Training in chennai

angularjs Training in chennai

angularjs-Training in tambaram

angularjs-Training in sholinganallur

ganga said...

Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.

angularjs Training in bangalore

angularjs Training in bangalore

angularjs Training in btm

angularjs Training in electronic-city

angularjs online Training

angularjs Training in marathahalli

Unknown said...

myTectra Placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance

Preethi Shetty said...

Needed to compose one simple word yet thanks for the suggestions that you are contributed here, please do keep updating us...
Data Science Training | Data Science Online Training

sandy star said...

Does your blog have a contact page? I’m having problems locating it but, I’d like to shoot you an email. I’ve got some recommendations for your blog you might be interested in hearing.
AWS Training in Chennai |Best Amazon Web Services Training in Chennai
AWS Training in Rajaji Nagar | Amazon Web Services Training in Rajaji Nagar

Best AWS Training Institute in BTM Layout Bangalore ,AWS Coursesin BTM

Web Bila said...

Website designer in Chandigarh
SEO company in chandigarh
Digital Marketing company in chandigarh