How to drain/delete/expire existing messages in Kafka

Some time you might have a bad record in Kafka topic that you want to delete. Kafka does not provide direct option to delete specific record. Only way to delete records is to expire them. You can achieve this by setting data retention to say 1 second that expires all the old messages. You can follow these steps
    First check the topic to find out value of retention.ms config parameter for the topic
  1. 
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name my-topic --entity-type topics
    Configs for topic 'my-topic' are retention.ms=86400000
    
  2. Change value of retention.ms to 1 which means all messages older than 1 ms will be expired
    
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name my-topic --entity-type topics --add-config retention.ms=1
    Completed Updating config for entity: topic 'my-topic'
    
  3. Wait for few seconds and monitor logs of the Kafka server to make sure that messages from the topic are deleted
  4. Now change the value of retention.ms back to its original value which was 86400000 (7 days)
    
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --alter --entity-name my-topic --entity-type topics --add-config retention.ms=86400000
    Completed Updating config for entity: topic 'my-topic'.
    
  5. You can verify that your changes are saved by running this command
    
    kafka_2.11-0.11.0.1 spatil$ bin/kafka-configs.sh --zookeeper localhost:2181 --describe --entity-name my-topic --entity-type topics
    Configs for topic 'my-topic' are retention.ms=86400000
    

263 comments:

«Oldest   ‹Older   201 – 263 of 263
Bygrad Education said...

Learn advanced Data Science & Machine Learning skills from world-renowned IBM faculty. Learn Machine Learning, Deep Learning, NLP, AI, Recommendation Systems, and more. Apply Now. Real-world Case Studies. Access to resource lab. Hands-on Projects.
Apply now: Online Data Science PG Program

Life coach said...

Great Post... Need certification in life coach career. Read more..

Matrrix is a premium Life Coaching and Training institute (India) established in 2015. The firm offers the ‘Become an Associate Coach’ (BAC) program, and the ‘Become a Professional Coach’ (BPC) program for aspiring coaching professionals.
In this Life Coaching Certification program, learners will have a deep understanding of ICF Core Competencies with additional knowledge of the application of prominent psychology techniques and training models that are useful to gain a higher level of awareness of human behavior.
Dr. Paras has devised exclusive coach supervision and mentoring programs aimed to help all professionals sharpen their skills in the field of life coaching.

Life coach said...

Great post

Life coach said...

Keep writing... Its an amazing post!!

Matrrix is a premium Life Coaching and Training institute (India) established in 2015. The firm offers the ‘Become an Associate Coach’ (BAC) program, and the ‘Become a Professional Coach’ (BPC) program for aspiring coaching professionals.
In this Life Coaching Certification program, learners will have a deep understanding of ICF Core Competencies with additional knowledge of the application of prominent psychology techniques and training models that are useful to gain a higher level of awareness of human behavior.
Dr. Paras has devised exclusive coach supervision and mentoring programs aimed to help all professionals sharpen their skills in the field of life coaching.

sanket said...
This comment has been removed by the author.
360DigiTMGAurangabad said...

Hi Thanks for Sharing this Valuable Information with us: this is very useful for me. Keep it Up.
data scientist courses in aurangabad

Citywebpuneind said...

Very interesting information I like this blog very much so much superb information
Web Development Pune

Citywebpuneind said...
This comment has been removed by the author.
ndteche said...

Exellent your post. i reaching daily on your blogs. thanks for sharing this informative information.
Buy YouTube Views
Buy YouTube Subscribers
Buy YouTube Live Stream Views
Facebook Live Views
Free Instagram Reels Views
Instagram Live Views
Instagram Story Views
Facebook Views

Meghana said...

Ternary Materials Market 2022 Examination and Industry Growth With Top Companies, Regions till 2028
Summary

A New Market Study, Titled “Ternary Materials Market Upcoming Trends, Growth Drivers and Challenges” has been featured on fusionmarketresearch.

This report provides in-depth study of ‘Ternary Materials Market ‘using SWOT analysis i.e. strength, weakness, opportunity and threat to Organization. The Ternary Materials Market report also provides an in-depth survey of major market players which is based on the various objectives of an organization such as profiling, product outline, production quantity, raw material required, and production. The financial health of the organization.

Ternary Materials Market

Meghana said...

Textile Auxiliaries Market Size 2022 | Industry Share, Future Growth, Business, Investment Opportunities, Covid-19 Impact, Key Manufacturers and 2028
Summary
A New Market Study, Titled “Textile Auxiliaries Market Upcoming Trends, Growth Drivers and Challenges” has been featured on fusionmarketresearch.

This report provides in-depth study of ‘Textile Auxiliaries Market ‘using SWOT analysis i.e. strength, weakness, opportunity and threat to Organization. The Textile Auxiliaries Market report also provides an in-depth survey of major market players which is based on the various objectives of an organization such as profiling, product outline, production quantity, raw material required, and production. The financial health of the organization.

Textile Auxiliaries Market

James said...

Mineral Wool Market Status (2016-2020) and Forecast (2021E-2028F) by Region, Product Type & End-Use
Mineral Wool market

Overview

At the beginning of a recently published report on the global Mineral Wool market, extensive analysis of the industry has been done with an insightful explanation. The overview has explained the potential of the market and the role of key players that have been portrayed in the information that revealed the applications and manufacturing technology required for the growth of the global Mineral Wool market.

Mineral Wool market

fonadaivr said...


its very nice blog for mecustomer service!

Anonymous said...
This comment has been removed by the author.
Pharma Online said...

cheap tramadol 100mg online

prem kumar said...

cheap tramadol 100mg online
Purchased Now 1-315-512-2220, Buy Tramadol Online 100Mg Pills Without Prescription at lowest cost from the licensed supplier.

prem kumar said...

buy tramadol 150mg online

prem kumar said...

buy tramadol 225mg online
If you are looking for Buy Cheap 100mgTramadol Online in USA, Then you are at the right place. The Online Pharmacy USA is the best place to buy tramadol online.

prem kumar said...

buy tramadol 225mg online
Buy Tramadol Online in USA. Tramadol 50mg,100mg,150mg,200mg,225mg. Brand Name: OL-TRAM / ULTRAM. Order Tramadol from our website with affordable price rates.

vivek kumar said...

Talk to Best Astrologer
AstrologyKart is the best astrology website for online astrology predictions from the best astrologers of India. Chat with an astrologer in India online & also talk to best experienced astrologers online anytime at AstrologyKart.

vivek kumar said...

Free Astrology Advice online
AstrologyKart is the best astrology website for online astrology predictions from the best astrologers of India. Chat with an astrologer in India online & also talk to best experienced astrologers online anytime at AstrologyKart.

prem kumar said...

Mobile tower installation in Chhattisgarh
Providing the top level mobile tower installation in Chhattisgarh with Digital Groups. We are delivering the Mobile Tower Installation in rural as well as urban areas of Chhatisgarh.

prem kumar said...

Mobile tower installation in Bihar
Are you looking for the best mobile tower installation company in Bihar? Call now 7864805461 for Applying for Mobile Tower Installation in Bihar, Patna, Bhagalpur.

vivek kumar said...

Best Ayurvedic Treatment
Ayurvedic Hospital in Dhanbad
Call Us at +91 9204900900, Kerala Ayurveda Dhanbad is one of the Best Ayurvedic Hospital, Providing Unbeatable Ayurvedic Treatments in Dhanbad.

vivek kumar said...

Best Ayurvedic Treatment Ranchi
Our Best Ayurvedic doctors in Ranchi are there to help you. Kerala Ayurveda Ranchi Also provides the best Ayurvedic infertility treatment in Ranchi.

vivek kumar said...

Best Ayurvedic Treatment in Bariatu
We provide the best Ayurvedic treatment in Bariatu, Kerala Ayurveda Ranchi is specialized in traditional ayurvedic treatment for diseases. Visit our website for more details.

PMP Training in Malaysia said...

360DigiTMG, the top-rated organisation among the most prestigious industries around the world, is an educational destination for those looking to pursue their dreams around the globe. The company is changing careers of many people through constant improvement, 360DigiTMG provides an outstanding learning experience and distinguishes itself from the pack. 360DigiTMG is a prominent global presence by offering world-class training. Its main office is in India and subsidiaries across Malaysia, USA, East Asia, Australia, Uk, Netherlands, and the Middle East.

360digiTMG.com said...

New site is solid. A debt of gratitude is in order for the colossal exertion. business analytics course in mysore

data analytics courses malaysia said...

I am another customer of this site so here I saw various articles and posts posted by this site,I curious more energy for some of them trust you will give more information further. data analytics courses malaysia

data analyst course malaysia said...

Through this post, I realize that your great information in playing with all the pieces was exceptionally useful. I advise this is the primary spot where I discover issues I've been scanning for. You have a smart yet alluring method of composing.data analyst course malaysia

said...

It is very nice website with all proper information and also get the best eCommerce web design Dubai

Moti Mahal Group said...

Thanks for Sharing. It's a very informative blog. Moti Mahal Group is one of the most prestigious names in food and hospitality industry which has many restaurants and eating joints in India and across the world. You can also become the valuable partner of this esteemed organization by taking its franchise and own Moti Mahal delux restaurant in your area or city. Read on to know how to start Moti Mahal restaurant in India, cost of starting Moti Mahal restaurant and where to contact for getting Moti Mahal franchise in India.

Idams said...

Good article.
ecommerce photo editing service

data science course in borivali said...

This article will present a closer look at data science courses that give you a comprehensive look at the field. So let's get started.
data science course in borivali

Swarnalatha said...

"If you are also one of them and want to know what the companies demand from the data scientists to do in their organization, you have come to the right place.data science course in kolkata"

Anonymous said...

It's an excellent article, mind-blowing … if you are looking for more information on dedicated server hosting, check out an Malaysia VPS Server Hosting, which is unlike anything else on the market.You Should write on Malaysia VPS Hosting it offers an amazing hosting solution to users.

Anonymous said...

This article was very well written by you. It's exact and of excellent quality. It is readable and simple to read thanks to your efforts. You really are a wonderful writer. Thank you a lot. For more information then click here USA VPS Server

Anonymous said...

Lovely blog; thanks for letting us know about this. If you are searching for the fastest and cheap USA dedicated server, you can ask us for more details and services.. USA Dedicated Server

shiva pal said...

Click on sites to list free classifieds
https://addfreepost.com/
https://web.addfreepost.com/
https://geo.addfreepost.com/
https://freeadgreen.com/
https://adsbags.com/
https://h5ad.org/
https://trumasterad.com/

deekshitha said...

Python is a programming language that enables quicker work and more efficient system integration. Because of its simplicity, ease of use, and accessibility to libraries like NumPy, Pandas, and Matplotlib, it is one of the best languages used by data scientists for a variety of data science applications. You will gain a multidisciplinary skill set and be prepared to work with massive volumes of data to find insights and solutions to business challenges after completing this data science certification course. Having this certification will provide you a technical understanding of computer science and statistics in addition to a postgraduate degree. It covers the spring framework, hibernate, and advanced java. data science course institute in nagpur

Anonymous said...

Thank you, got to know a lot from this blog, thank you for this information. If you are searching for fastest and cheap Ukraine Dedicated server, you can ask us for more details and services. Ukraine Dedicated Server

divya"s fashion said...

Thanks for sharing,RR Technosoft is the best azure devopsTraining institute in Hyderabad and it provides Class room & Online Training by real time faculty with course material and Lab Facility.


Pet Spoiler said...

Thanks for sharing such a valuable information.
Rabbit grooming Shop
Rabbit grooming Shop dubai

professional movers and packers said...

Good article.
professional movers and packers in dubai
professional movers dubai

Professional Movers and packers in Abu dhabi said...

Nice effort.
local movers in abu dhabi
professional furniture movers

Ruhi said...

Thanks For Sharing The Information.Algorithmic Excellence Choosing the Right Algorithms for Python Optimization

Chaitanya said...

This is my first time go to see at here and i am truly happy to read all at single place.
SSAS Online Training
Oracle Fusion Cloud CRM Online Training
Looker Online Training
Arcsight Online Training

vcube said...

Nice Post keep posting.
Best Python Training institute in Hyderabad

Harish said...

Hi Sunil, I am not sure about India but in the United States, there is a huge demand shift towards artificial intelligence technology instead of website developers. That means businesses are using AI tools and technologies to replace website developers. So, I would suggest you to understand more about the impacts that AI can have on the website development industry.

syntaxminds said...

Really helpful article thank you for sharing. Data science training in Hyderabad

sravyaa said...

very nice blog
azure devops training in hyderabad/

arjuna said...

nice blog
https://nagarjuna.online/

vishwa said...

good information
Digital Marketing

yesfinancemumbai said...

great post

Personal loans in Himatnagar

sathvik nayak said...

The depth and clarity of this article are impressive. You’ve made a potentially confusing topic feel organized and approachable.
Data science courses in Noida

Data Science said...

To remove unwanted messages from a Kafka topic, setting retention.ms to a very low value (e.g., 1 ms) is an effective workaround, as Kafka does not support deleting specific records directly. This method expires all older messages, effectively clearing the topic. Remember to reset retention.ms to its original value afterward to restore standard data retention. Data science courses in Gurgaon

Sadhvi said...

This article raises interesting points about the dynamics between Turkey, Russia, and the U.S. in northeast Syria. Data science courses in Visakhapatnam

Rachana said...


This blog provides a clear and practical guide to expiring unwanted messages in Kafka topics using the retention.ms parameter. It's a useful resource for handling such scenarios effectively. Great work!
Data science courses in Gujarat

sakshi.gupta.university said...

Kafka's message retention and deletion policies are essential for managing data effectively. By configuring retention times or deleting topics, you can control how long messages persist. It's important to adjust these settings based on your use case to optimize storage and performance.

Data science courses in Pune

P. Zaheer Khan said...

Thank you for this fantastic guide! Your approach to Kafka message deletion is concise and effective. Looking forward to more such posts.
Data science Courses in Sydney

IIM Skills Data Science said...

"Thanks for the clear explanation on managing data retention! The step-by-step approach is very helpful for handling existing logs effectively."
Data science Courses in Canada

kriti sharma said...

This is a very detailed guide on managing WebSphere sessions effectively! It's incredibly helpful for anyone working with large-scale applications that require fine-tuned session management. I appreciate the step-by-step instructions, and the emphasis on cleaning up expired sessions to optimize system performance. I’m sure this will save me a lot of time when troubleshooting session issues
Data science courses in glasgow

NEEL KBH said...

Managing Kafka topics efficiently is crucial, and understanding retention policies can save the day. This step-by-step guide on expiring messages is a lifesaver for clearing unwanted data without impacting topic integrity. A practical solution for maintaining Kafka’s performance and reliability!

Data science Courses in City of Westminster

Neel KBH
kbhneel@gmail.com

«Oldest ‹Older   201 – 263 of 263   Newer› Newest»