You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document
Spark Streaming Kafka 10 API Word Count application Scala
In Spark Kafka Streaming Java program Word Count using Kafka 0.10 API blog entry i talked about how you create a simple java program that uses Spark Streaming's Kafka10 API using Java. This blog entry does the same thing but using Scala. You can download the complete application from github
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document
dear sunil,
ReplyDeleteI am new and I am using Intlej to get the code working. I need the build.sbt file instead of pom.xml, as I am using sbt.
spent hours and hours but not able to get a complete project up and running. Is there a way you can help me out.
Steps like in intlej select import or new project of if you can provide me a starting point.
I have kafka and zookeeper running fine on my windows machine.
Appreciate your help
Getting error Exception in thread "main" java.lang.ClassNotFoundException: Kafka10
ReplyDeleteHi, i’m Developer a test using spark streaming consumer from Kafka topic. I used this program, but when start the program erro to connect localhost 9999 and I need start from console nc -lk 9999 but I want consumer from topic.
ReplyDeletevery informative blog and useful article thank you for sharing with us, keep posting Big data hadoop online training India
ReplyDeleteHi,
ReplyDeleteI tried your above example in intellij with details below:
name := "WORDCOUNT"
version := "0.1"
scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"
the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.
Hi,
ReplyDeleteI tried your above example in intellij with details below:
name := "WORDCOUNT"
version := "0.1"
scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"
the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.
Kindly let me know way to solve it
Java > Java2D Code Examples
ReplyDeletejava programming with samples
Thanks for sharing valuable Information, I really very impressive on your blog. I hope you continue on blogging job.
ReplyDeleteI have a hard time describing my thoughts on content, but I really felt I should here. Your article is really great. I like the way you wrote this information. read
ReplyDeleteIf you are facing any problem with YouTube not working your device like smart TV or Roku device etc. so you can contact us, Our YouTube Activate customer care team 24*7 available for resolving your issues, furthermore gives the safest solution by simple steps.
ReplyDeleteThank you for taking the time to publish this information very useful!
ReplyDeleteaws solutions architect certification
ReplyDeleteThis blog is so nice to me. I will keep on coming here again and again. Visit my link as well..
aws training
I appreciated your work very thanks
ReplyDeleteaws course
ReplyDeleteThis is such a great resource that you are providing and you give it away for free.
business analysis certification
ReplyDeleteYes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!THANKS!!!!!!
itil foundation
All the contents you mentioned in post is too good and can be very useful. I will keep it in mind, thanks for sharing the information keep updating, looking forward for more posts.Thanks free word counter
ReplyDeletethanks to share spark streaming example in simple way.
ReplyDeleteRegards
Venu
bigdata training institute in Hyderabad
spark training in Hyderabad
Thanks for Information
ReplyDeleteDigital Marketing Services - Nikke Tech