Spark Streaming Kafka 10 API Word Count application Scala

In Spark Kafka Streaming Java program Word Count using Kafka 0.10 API blog entry i talked about how you create a simple java program that uses Spark Streaming's Kafka10 API using Java. This blog entry does the same thing but using Scala. You can download the complete application from github
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document

19 comments:

Anonymous said...

dear sunil,
I am new and I am using Intlej to get the code working. I need the build.sbt file instead of pom.xml, as I am using sbt.

spent hours and hours but not able to get a complete project up and running. Is there a way you can help me out.
Steps like in intlej select import or new project of if you can provide me a starting point.
I have kafka and zookeeper running fine on my windows machine.

Appreciate your help

Anonymous said...

Getting error Exception in thread "main" java.lang.ClassNotFoundException: Kafka10

Unknown said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Spark and Scala, kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Sangita Mohanty
MaxMunus
E-mail: sangita@maxmunus.com
Skype id: training_maxmunus
Ph:(0) 9738075708 / 080 - 41103383
http://www.maxmunus.com/

Tiago Ferrucio said...

Hi, i’m Developer a test using spark streaming consumer from Kafka topic. I used this program, but when start the program erro to connect localhost 9999 and I need start from console nc -lk 9999 but I want consumer from topic.

Tejuteju said...

very informative blog and useful article thank you for sharing with us, keep posting Big data hadoop online training India

Unknown said...

Hi,

I tried your above example in intellij with details below:
name := "WORDCOUNT"

version := "0.1"

scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"

the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.

Unknown said...

Hi,

I tried your above example in intellij with details below:
name := "WORDCOUNT"

version := "0.1"

scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"

the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.

Kindly let me know way to solve it

xcod4r said...

Java > Java2D Code Examples
java programming with samples

Mobile App Development company in Mumbai said...

Thanks for sharing valuable Information, I really very impressive on your blog. I hope you continue on blogging job.

Anonymous said...

I have a hard time describing my thoughts on content, but I really felt I should here. Your article is really great. I like the way you wrote this information. read

youtubecomactivate said...

If you are facing any problem with YouTube not working your device like smart TV or Roku device etc. so you can contact us, Our YouTube Activate customer care team 24*7 available for resolving your issues, furthermore gives the safest solution by simple steps.

unknown said...

Thank you for taking the time to publish this information very useful!


aws solutions architect certification

unknown said...


This blog is so nice to me. I will keep on coming here again and again. Visit my link as well..

aws training

unknown said...

I appreciated your work very thanks


aws course

unknown said...


This is such a great resource that you are providing and you give it away for free.

business analysis certification

unknown said...


Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!THANKS!!!!!!

itil foundation

Jasmine Dale said...

All the contents you mentioned in post is too good and can be very useful. I will keep it in mind, thanks for sharing the information keep updating, looking forward for more posts.Thanks free word counter

Sreyobhilashi Institute said...

thanks to share spark streaming example in simple way.
Regards
Venu
bigdata training institute in Hyderabad
spark training in Hyderabad

Google Blog said...

Thanks for Information

Digital Marketing Services - Nikke Tech