Spark Streaming Kafka 10 API Word Count application Scala

In Spark Kafka Streaming Java program Word Count using Kafka 0.10 API blog entry i talked about how you create a simple java program that uses Spark Streaming's Kafka10 API using Java. This blog entry does the same thing but using Scala. You can download the complete application from github
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document

20 comments:

Anonymous said...

dear sunil,
I am new and I am using Intlej to get the code working. I need the build.sbt file instead of pom.xml, as I am using sbt.

spent hours and hours but not able to get a complete project up and running. Is there a way you can help me out.
Steps like in intlej select import or new project of if you can provide me a starting point.
I have kafka and zookeeper running fine on my windows machine.

Appreciate your help

Anonymous said...

Getting error Exception in thread "main" java.lang.ClassNotFoundException: Kafka10

Dattatray Bhosale said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Spark and Scala, kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Sangita Mohanty
MaxMunus
E-mail: sangita@maxmunus.com
Skype id: training_maxmunus
Ph:(0) 9738075708 / 080 - 41103383
http://www.maxmunus.com/

sunitha vishnu said...


This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information&its very useful to me...
Android training in chennai
Ios training in chennai

rithkhmer said...

Thank you very much for sharing such a beautiful article.
หนังออนไลน์

Jason Borne said...

We are a third party technical support service. Avast Customer Support is here to help you out with the whole procedure to Download Avast Antivirus online, We not only fix your Avast Support related issues but will guide with how to get started with your new Avast product once it gets installed successfully. Call on our Toll Free no. 1 855 966 3855
Gmail Customer service is a third party technical support service for Gmail users when they face any technical issue or error in their Gmail account. Our Gmail Customer Support team solves issues like forgot Gmail account password, Gmail configuration or Sync issues, recover deleted emails and many more. Toll Free number (800) 986-9271
How you install or reinstall Office 365 or Office 2016 depends on whether your Office product is part of an Office for home or Office for business plan. If you're not sure what you have, see what office com setup products are included in each plan and then follow the steps for your product. The steps below also apply if you're installing a single, stand-alone Office application such as Access 2016 or Visio 2016. Need Help with office setup Enter Product Key? Call 1-800-000-0000 Toll Free
Norton Tech Support is a third party service provider and not in any way associated with Norton or any of its partner companies. We offer support for Norton products and sell subscription based additional warranty on computer and other peripheral devices. Call our Toll Free number 1 855 966 3855
Other Services
Norton Toll Free , Office-Setup , office.com/setup.

sangi yadav said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in APACHE SPARK , kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training On APACHE SPARK . We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Saurabh Srivastava
MaxMunus
E-mail: saurabh@maxmunus.com
Skype id: saurabhmaxmunus
Ph:+91 8553576305 / 080 - 41103383
http://www.maxmunus.com/


sangi yadav said...

I really appreciate information shared above. It’s of great help. If someone wants to learn Online (Virtual) instructor lead live training in APACHE KAFKA , kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training On APACHE KAFKA. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Saurabh Srivastava
MaxMunus
E-mail: saurabh@maxmunus.com
Skype id: saurabhmaxmunus
Ph:+91 8553576305 / 080 - 41103383
http://www.maxmunus.com/


Philips Huges said...

Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

Installment Loans Near Me
Title loans Near Me
Cash Advances Near Me

Tiago Ferrucio said...

Hi, i’m Developer a test using spark streaming consumer from Kafka topic. I used this program, but when start the program erro to connect localhost 9999 and I need start from console nc -lk 9999 but I want consumer from topic.

srjwebsolutions said...


We are leading responsive website designing and development company in Noida.
We are offering mobile friendly responsive website designing, website development, e-commerce website, seo service and sem services in Noida.

Responsive Website Designing Company in Noida
Website Designing Company in Noida
SEO Services in Noida
SMO Services in Noida

Vikas Chaudhary said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

EG MEDI said...

Egmedi.com is online medical store pharmacy in laxmi nagar Delhi. You can Order prescription/OTC medicines online. Cash on Delivery available. Free Home Delivery


Online Pharmacy in Delhi
Buy Online medicine in Delhi
Online Pharmacy in laxmi nagar
Buy Online medicine in laxmi nagar
Onine Medical Store in Delhi
Online Medical store in laxmi nagar
Online medicine store in delhi
online medicine store in laxmi nagar
Purchase Medicine Online
Online Pharmacy India
Online Medical Store

Teju Teju said...

very informative blog and useful article thank you for sharing with us, keep posting Big data hadoop online training India

viswanath said...

AWS Training in Bangalore - Live Online & Classroom
myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

IOT Training in Bangalore - Live Online & Classroom
IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

Unknown said...

Hi,

I tried your above example in intellij with details below:
name := "WORDCOUNT"

version := "0.1"

scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"

the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.

Unknown said...

Hi,

I tried your above example in intellij with details below:
name := "WORDCOUNT"

version := "0.1"

scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"

the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.

Kindly let me know way to solve it

Unknown said...

myTectra Placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance

DJ PRASATH said...

Thanks for your post. This is excellent information. The list of your blogs is very helpful for those who want to learn, It is amazing!!! You have been helping many application.
best selenium training in chennai | best selenium training institute in chennai selenium training in chennai | best selenium training in chennai | selenium training in Velachery | selenium training in chennai omr | quora selenium training in chennai | selenium testing course fees | java and selenium training in chennai | best selenium training institute in chennai | best selenium training center in chennai

xcod4r said...

Java > Java2D Code Examples
java programming with samples