You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document
Spark Streaming Kafka 10 API Word Count application Scala
In Spark Kafka Streaming Java program Word Count using Kafka 0.10 API blog entry i talked about how you create a simple java program that uses Spark Streaming's Kafka10 API using Java. This blog entry does the same thing but using Scala. You can download the complete application from github
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document
You can run this sample by first downloading Kafka 0.10.* from Apache Kafka WebSite, then you can create and start a test topic and send messages to it by following this Kafka Quick start document
Subscribe to:
Post Comments (Atom)
18 comments:
dear sunil,
I am new and I am using Intlej to get the code working. I need the build.sbt file instead of pom.xml, as I am using sbt.
spent hours and hours but not able to get a complete project up and running. Is there a way you can help me out.
Steps like in intlej select import or new project of if you can provide me a starting point.
I have kafka and zookeeper running fine on my windows machine.
Appreciate your help
Getting error Exception in thread "main" java.lang.ClassNotFoundException: Kafka10
Hi, i’m Developer a test using spark streaming consumer from Kafka topic. I used this program, but when start the program erro to connect localhost 9999 and I need start from console nc -lk 9999 but I want consumer from topic.
very informative blog and useful article thank you for sharing with us, keep posting Big data hadoop online training India
Hi,
I tried your above example in intellij with details below:
name := "WORDCOUNT"
version := "0.1"
scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"
the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.
Hi,
I tried your above example in intellij with details below:
name := "WORDCOUNT"
version := "0.1"
scalaVersion := "2.12.6"
val sparkVersion = "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.3"
libraryDependencies += "org.apache.spark" % "spark-streaming-kafka-0-10_2.10" % "2.0.0"
scalacOptions += "-Ylog-classpath"
the same program i am using with error as below:
Error:(9, 8) Symbol 'type org.apache.spark.internal.Logging' is missing from the classpath.
This symbol is required by 'object org.apache.spark.streaming.kafka010.KafkaUtils'.
Make sure that type Logging is in your classpath and check for conflicting dependencies with `-Ylog-classpath`.
A full rebuild may help if 'KafkaUtils.class' was compiled against an incompatible version of org.apache.spark.internal.
import scala.collection.mutable.
Kindly let me know way to solve it
Java > Java2D Code Examples
java programming with samples
Thanks for sharing valuable Information, I really very impressive on your blog. I hope you continue on blogging job.
I have a hard time describing my thoughts on content, but I really felt I should here. Your article is really great. I like the way you wrote this information. read
If you are facing any problem with YouTube not working your device like smart TV or Roku device etc. so you can contact us, Our YouTube Activate customer care team 24*7 available for resolving your issues, furthermore gives the safest solution by simple steps.
Thank you for taking the time to publish this information very useful!
aws solutions architect certification
This blog is so nice to me. I will keep on coming here again and again. Visit my link as well..
aws training
I appreciated your work very thanks
aws course
This is such a great resource that you are providing and you give it away for free.
business analysis certification
Yes i am totally agreed with this article and i just want say that this article is very nice and very informative article.I will make sure to be reading your blog more. You made a good point but I can't help but wonder, what about the other side? !!!!!!THANKS!!!!!!
itil foundation
All the contents you mentioned in post is too good and can be very useful. I will keep it in mind, thanks for sharing the information keep updating, looking forward for more posts.Thanks free word counter
thanks to share spark streaming example in simple way.
Regards
Venu
bigdata training institute in Hyderabad
spark training in Hyderabad
Thanks for Information
Digital Marketing Services - Nikke Tech
Post a Comment