-
First i changed the workflow.xml to take out
as-avrodatafileand addedhive-importoption and i re-ran the workflow that looks like this When i did that the oozie workflow failed with following error7936 [uber-SubtaskRunner] WARN org.apache.sqoop.mapreduce.JobBase - SQOOP_HOME is unset. May not be able to find all job dependencies. 9202 [uber-SubtaskRunner] DEBUG org.apache.sqoop.mapreduce.db.DBConfiguration - Fetching password from job credentials store 9207 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.db.DBInputFormat - Using read commited transaction isolation 9210 [uber-SubtaskRunner] DEBUG org.apache.sqoop.mapreduce.db.DataDrivenDBInputFormat - Creating input split with lower bound '1=1' and upper bound '1=1' 25643 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.ImportJobBase - Transferred 931.1768 KB in 17.6994 seconds (52.6107 KB/sec) 25649 [uber-SubtaskRunner] INFO org.apache.sqoop.mapreduce.ImportJobBase - Retrieved 12435 records. 25649 [uber-SubtaskRunner] DEBUG org.apache.sqoop.hive.HiveImport - Hive.inputTable: customers 25650 [uber-SubtaskRunner] DEBUG org.apache.sqoop.hive.HiveImport - Hive.outputTable: customers 25653 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Execute getColumnInfoRawQuery : SELECT t.* FROM `customers` AS t LIMIT 1 25653 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - No connection paramenters specified. Using regular API for making connection. 25658 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Using fetchSize for next query: -2147483648 25658 [uber-SubtaskRunner] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1 25659 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_id of type [4, 11, 0] 25659 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_fname of type [12, 45, 0] 25659 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_lname of type [12, 45, 0] 25660 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_email of type [12, 45, 0] 25660 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_password of type [12, 45, 0] 25660 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_street of type [12, 255, 0] 25660 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_city of type [12, 45, 0] 25660 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_state of type [12, 45, 0] 25660 [uber-SubtaskRunner] DEBUG org.apache.sqoop.manager.SqlManager - Found column customer_zipcode of type [12, 45, 0] 25663 [uber-SubtaskRunner] DEBUG org.apache.sqoop.hive.TableDefWriter - Create statement: CREATE TABLE IF NOT EXISTS `customers` ( `customer_id` INT, `customer_fname` STRING, `customer_lname` STRING, `customer_email` STRING, `customer_password` STRING, `customer_street` STRING, `customer_city` STRING, `customer_state` STRING, `customer_zipcode` STRING) COMMENT 'Imported by sqoop on 2016/12/22 21:18:39' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE 25664 [uber-SubtaskRunner] DEBUG org.apache.sqoop.hive.TableDefWriter - Load statement: LOAD DATA INPATH 'hdfs://quickstart.cloudera:8020/user/cloudera/customers' INTO TABLE `customers` 25667 [uber-SubtaskRunner] INFO org.apache.sqoop.hive.HiveImport - Loading uploaded data into Hive 25680 [uber-SubtaskRunner] DEBUG org.apache.sqoop.hive.HiveImport - Using in-process Hive instance. 25683 [uber-SubtaskRunner] DEBUG org.apache.sqoop.util.SubprocessSecurityManager - Installing subprocess security manager Intercepting System.exit(1) <<< Invocation of Main class completed <<< Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] Oozie Launcher failed, finishing Hadoop job gracefully Oozie Launcher, uploading action data to HDFS sequence file: hdfs://quickstart.cloudera:8020/user/cloudera/oozie-oozi/0000007-161222163830473-oozie-oozi-W/sqoop-52c0--sqoop/action-data.seq Oozie Launcher ends -
As you can see from the log the Sqoop job was able to import data into HDFS in
/user/cloudera/customersdirectory and i could actually see the data in the directory. But when Sqoop tried to create the table in hive it failed and the table did not get created in hive, this is the log statement that i am referring toCREATE TABLE IF NOT EXISTS `customers` ( `customer_id` INT, `customer_fname` STRING, `customer_lname` STRING, `customer_email` STRING, `customer_password` STRING, `customer_street` STRING, `customer_city` STRING, `customer_state` STRING, `customer_zipcode` STRING) COMMENT 'Imported by sqoop on 2016/12/22 21:18:39' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE - So it seems the problem is Sqoop needs hive-site.xml so that it knows how to talk to hive service, for that first i search my sandbox to figure out where hive-site.xml is located, i executed following command to first find the hive-site.xml and then uploading it to HDFS
sudo find / -name hive-site.xml hdfs dfs -put /etc/hive/conf.dist/hive-site.xml - After that i went back to the workflow.xml and modified it to look like this
Importing data from RDBMS into Hive using Sqoop and oozie (hive-import)
In the How to run Sqoop command from oozie entry i talked about how you can use Oozie and Sqoop to import data into HDFS. I wanted to change it to use sqoop's hive-import option, which in addition to importing data into HDFS also creats Hive table on top of the data. These are the steps that i followed
How to run Sqoop command from oozie
In the Importing data from Sqoop into Hive External Table with Avro encoding updated i blogged about how you can use sqoop to import data from RDBMS into Hadoop. I wanted to test if i can use Oozie for invoking Sqoop command and i followed these steps for doing that.
-
First i tried executing this command from my command line on Hadoop cluster to make sure that i can actually run sqoop without any problem
sqoop import --connect jdbc:mysql://localhost/test --username root --password cloudera --table CUSTOMER --as-avrodatafile -
Once the sqoop command was successfully executed i went back and deleted the CUSTOMER directory from HDFS to make sure that i could re-import data using following command
hdfs dfs -rm -R CUSTOMER -
Next i went to Hue to create oozie workflow with single sqoop command that i had executed before
But if your not using the Hue console you can create workflow.xml manually like this Also make sure to create job.properties file like this Take a look at Enabling Oozie console on Cloudera VM 4.4.0 and executing examples for information on how to run oozie job from command line
- Next when i ran the Oozie workflow, the job failed with following error, which indicates that Oozie does not have the MySQL JDBC driver.
java.lang.RuntimeException: Could not load db driver class: com.mysql.jdbc.Driver at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:875) at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:52) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763) at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1846) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1646) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49) - So first thing i did was to check if mysql driver is there in the oozie shared lib by executing following commands
I noticed that the mysql-connector-java.jar was not there in the list of shared libs for Oozie + sqoopexport OOZIE_URL=http://localhost:11000/oozie oozie admin -shareliblist sqoop - Next step was to find the mysql-connector-java.jar in my sandbox that i could do by finding it like this
I found mysql-connector-java.jar on my local machine atsudo find / -name mysql*/var/lib/sqoop/mysql-connector-java.jar - I wanted to update the Oozie shared lib to include the mysql driver jar. So i executed following command to figure out the directory where the oozie sqoop shared lib is
oozie admin -sharelibupdateFrom this output i got HDFS directory location for Oozie shared lib which is
/user/oozie/share/lib/lib_20160406022812 - Then i used following two commands to first copy the db driver into the oozie shared lib and making sure it is accessible to other users
hdfs -copyFromLocal /var/lib/sqoop/mysql-connector-java.jar /user/oozie/share/lib/sqoop/. hdfs dfs -chmod 777 /user/oozie/share/lib/sqoop/mysql-connector-java.jar - Now the last step was to let Oozie know that it should reload the sharedlib and i did that by executing following two commands
The second command queries oozie to get current list of shared jars and i could see mysql-connector-java.jar listed in it like thisoozie admin -sharedlibupdate oozie admin -shareliblist sqoop | grep mysql*
Importing data from Sqoop into Hive External Table with Avro encoding updated
In the Importing data from Sqoop into Hive External Table with Avro encoding i had details on how you can import a table from RDBMS into Hive using Sqoop in Avro format. In that blog i went through few steps to get the avsc file, but i realized there is easier way to do it following these steps
-
First execute the
sqoop importcommand like this, make sure that you pass--outdir schemaas parameters to the sqoop import command, what that does is it generates the CUSTOMER.avsc and CUSTOMER.java in the schema directory on your local machinesqoop import --connect jdbc:mysql://localhost/test --username root --password cloudera --table CUSTOMER --as-avrodatafile --outdir schema -
You can verify that CUSTOMER.avsc file got created as you expected by executing
ls -ltrA schema
-
Next create schema directory in HDFS by executing hdfs mkdir command like this
hdfs dfs -mkdir /user/cloudera/schema -
Copy the CUSTOMER.avsc from your local schema directory to HDFS in schema directory by executing following command
hdfs dfs -copyFromLocal schema/CUSTOMER.avsc /user/cloudera/schema/. -
Last step is to create Hive table with CUSTOMER.avsc as schema using following command
CREATE EXTERNAL TABLE CUSTOMER ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe' STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat' OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat' LOCATION '/user/cloudera/CUSTOMER' TBLPROPERTIES ('avro.schema.url'='/user/cloudera/schema/CUSTOMER.avsc');
Writing data from Spark to ElasticSearch
ElasticSearch for Apache Hadoop project has introduced a way to directly write to ElasticSearch without going through Elastic Search OutputFormat. I wanted to try that out so i built simple application that saves output of word count into Elastic Search, you can download this project from github
First thing that i had to do was to build maven pom.xml that includes
org.elasticsearch.elasticsearch-hadoop version 5.0 jar. I could not find it in the regular maven repository so i had to include elasticsearch repository in my pom.xml
Then this is how my Spark program looks like, the main part is line 42 where i create Map of all the properties that i need for saving this RDD into ElasticSearch and then line 43, where i am calling wordCountJson.saveToEs(esMap), which actually takes care of writing data into elasticsearch
How to use KafkaLog4jAppender for sending Log4j logs to kafka
Apache Kafka has a KafkaLog4jAppender that you can use for redirecting your Log4j log to Kafka topic. I wanted to try it out so i used following steps, you can download sample project from here
First i created a simple standalone java program that use Log4j like this. As you can see this is like any other normal Java program that uses Log4j.
Then in the log4j.properties file i added line 12 to 17 for using KafkaLog4jAppender, on line 13, value of brokerList property points to the Kafka server and line 14 value of topic points to the Kafka topic name to which logs should go.
Now before running this program make sure that you actually have topic named kafkalogger, if not you can create using this command
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic kafkalogger
bin/kafka-topics.sh --list --zookeeper localhost:2181
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic kafkalogger
WordCount program using Spark DataFrame
I wanted to figure out how to write Word Count Program using Spark DataFrame API, so i followed these steps.
Import 
Next split each of the line into words using split function. This will create a new DataFrame with words column, each words column would have array of words for that line
Next use explode transformation to convert the words array into a dataframe with word column. This is equivalent of using flatMap() method on RDD
Now you have data frame with each line containing single word in the file. So group the data frame based on word and count the occurrence of each word
This is the code you need if you want to figure out 20 top most words in the file
org.apache.spark.sql.functions._, it includes UDF's that i need to use
import org.apache.spark.sql.functions._
Create a data frame by reading README.md. When you read the file, spark will create a data frame with single column value, the content of the value column would be the line in the file
val df = sqlContext.read.text("README.md")
df.show(10,truncate=false)
val wordsDF = df.select(split(df("value")," ").alias("words"))
wordsDF.show(10,truncate=false)
val wordDF = wordsDF.select(explode(wordsDF("words")).alias("word"))
wordsDF.show(10,truncate=false)
val wordCountDF = wordDF.groupBy("word").count
wordCountDF.show(truncate=false)
wordCountDF.orderBy(desc("count")).show(truncate=false)
How to use built in spark UDF's
In the i talked about how to create a custom UDF in scala for spark. But before you do that always check Spark UDF's that are available with Spark already.
I have this sample Spark data frame with list of users
I wanted to sort the list of users in descending order of age so i used following 2 lines, first is to import functions that are available with Spark already and then i used 
Now if i wanted to sort the data frame records using age in ascending order
This is sample of how to use the 
desc function to order age in descending order
import org.apache.spark.sql.functions._
display(userDF.orderBy(desc("age")))
display(userDF.orderBy(asc("age")))
sum() function
userDF.select(sum("age")).show
How to use built in spark UDF's
In the i talked about how to create a custom UDF in scala for spark. But before you do that always check Spark UDF's that are available with Spark already.
I have this sample Spark data frame with list of users
I wanted to sort the list of users in descending order of age so i used following 2 lines, first is to import functions that are available with Spark already and then i used
Now if i wanted to sort the data frame records using age in ascending order
This is sample of how to use the
desc function to order age in descending order
import org.apache.spark.sql.functions._
display(userDF.orderBy(desc("age")))
display(userDF.orderBy(asc("age")))
sum() function
userDF.select(sum("age")).show
How to access Hive table from Spark in MapR sandbox
I was trying to figure out how to query a hive table from spark in MapR 5.1 sandbox . So i started spark-shell and tried to query the
sample_08 table and i got error saying no such table exists
scala> val sample08 = sqlContext.sql("select * from sample_08")
org.apache.spark.sql.AnalysisException: no such table sample_08; line 1 pos 14
at org.apache.spark.sql.catalyst.analysis.package$AnalysisErrorAt.failAnalysis(package.scala:42)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.getTable(Analyzer.scala:260)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$7.applyOrElse(Analyzer.scala:268)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$$anonfun$apply$7.applyOrElse(Analyzer.scala:264)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$resolveOperators$1.apply(LogicalPlan.scala:57)
at org.apache.spark.sql.catalyst.trees.CurrentOrigin$.withOrigin(TreeNode.scala:51)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:56)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan$$anonfun$1.apply(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$4.apply(TreeNode.scala:249)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.sql.catalyst.trees.TreeNode.transformChildren(TreeNode.scala:279)
at org.apache.spark.sql.catalyst.plans.logical.LogicalPlan.resolveOperators(LogicalPlan.scala:54)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:264)
at org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations$.apply(Analyzer.scala:254)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:83)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1$$anonfun$apply$1.apply(RuleExecutor.scala:80)
at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:111)
at scala.collection.immutable.List.foldLeft(List.scala:84)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:80)
at org.apache.spark.sql.catalyst.rules.RuleExecutor$$anonfun$execute$1.apply(RuleExecutor.scala:72)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.sql.catalyst.rules.RuleExecutor.execute(RuleExecutor.scala:72)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed$lzycompute(SQLContext.scala:932)
at org.apache.spark.sql.SQLContext$QueryExecution.analyzed(SQLContext.scala:932)
at org.apache.spark.sql.SQLContext$QueryExecution.assertAnalyzed(SQLContext.scala:930)
at org.apache.spark.sql.DataFrame.(DataFrame.scala:132)
at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51)
at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:741)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:19)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:24)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:26)
at $iwC$$iwC$$iwC$$iwC$$iwC.(:28)
at $iwC$$iwC$$iwC$$iwC.(:30)
at $iwC$$iwC$$iwC.(:32)
at $iwC$$iwC.(:34)
at $iwC.(:36)
at (:38)
at .(:42)
at .()
at .(:7)
at .()
at $print()
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:674)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
/opt/mapr/hive/hive-1.2/conf/hive-site.xml had hive.metastore.uris property pointing to thrift://localhost:9083, so i copied it in the hive-site.xml and restarted the shell. When i execute the same query i can see the results.
scala> val sample08 = sqlContext.sql("select * from sample_08")
sample08: org.apache.spark.sql.DataFrame = [code: string, description: string, total_emp: int, salary: int]
Subscribe to:
Posts (Atom)