Spark Streaming Kafka 10 API Word Count application Scala

In Spark Kafka Streaming Java program Word Count using Kafka 0.10 API blog entry i talked about how you create a simple java program that uses Spark Streaming's Kafka10 API using Java. This blog entry does the same thing but using Scala. You can download the complete application from github
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document

19 comments:

  1. dear sunil,
    I am new and I am using Intlej to get the code working. I need the build.sbt file instead of pom.xml, as I am using sbt.

    spent hours and hours but not able to get a complete project up and running. Is there a way you can help me out.
    Steps like in intlej select import or new project of if you can provide me a starting point.
    I have kafka and zookeeper running fine on my windows machine.

    Appreciate your help

    ReplyDelete
  2. Getting error Exception in thread "main" java.lang.ClassNotFoundException: Kafka10

    ReplyDelete
  3. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Spark and Scala, kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on TECHNOLOGY. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Sangita Mohanty
    MaxMunus
    E-mail: sangita@maxmunus.com
    Skype id: training_maxmunus
    Ph:(0) 9738075708 / 080 - 41103383
    http://www.maxmunus.com/

    ReplyDelete
  4. Hi, i’m Developer a test using spark streaming consumer from Kafka topic. I used this program, but when start the program erro to connect localhost 9999 and I need start from console nc -lk 9999 but I want consumer from topic.

    ReplyDelete
  5. very informative blog and useful article thank you for sharing with us, keep posting Big data hadoop online training India

    ReplyDelete
  6. Hi,

    I tried your above example in intellij with details below:
    name := "WORDCOUNT"

    version := "0.1"

    scalaVersion := "2.12.6"
    val sparkVersion = "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
    scalacOptions += "-Ylog-classpath"

    the same program i am using with error as below:
    Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
    This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
    Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
    A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
    import scala.collection.mutable.

    ReplyDelete
  7. Hi,

    I tried your above example in intellij with details below:
    name := "WORDCOUNT"

    version := "0.1"

    scalaVersion := "2.12.6"
    val sparkVersion = "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
    libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
    scalacOptions += "-Ylog-classpath"

    the same program i am using with error as below:
    Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
    This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
    Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
    A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
    import scala.collection.mutable.

    Kindly let me know way to solve it

    ReplyDelete
  8. Thanks for sharing valuable Information, I really very impressive on your blog. I hope you continue on blogging job.

    ReplyDelete
  9. I have a hard time describing my thoughts on content, but I really felt I should here. Your article is really great. I like the way you wrote this information. read

    ReplyDelete
  10. If you are facing any problem with YouTube not working your device like smart TV or Roku device etc. so you can contact us, Our YouTube Activate customer care team 24*7 available for resolving your issues, furthermore gives the safest solution by simple steps.

    ReplyDelete
  11. Thank you for taking the time to publish this information very useful!


    aws solutions architect certification

    ReplyDelete

  12. This blog is so nice to me. I will keep on coming here again and again. Visit my link as well..

    aws training

    ReplyDelete

  13. This is such a great resource that you are providing and you give it away for free.

    business analysis certification

    ReplyDelete

  14. Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!THANKS!!!!!!

    itil foundation

    ReplyDelete
  15. All the contents you mentioned in post is too good and can be very useful. I will keep it in mind, thanks for sharing the information keep updating, looking forward for more posts.Thanks free word counter

    ReplyDelete