Importing data from Sqoop into Hive External Table with Avro encoding updated

In the Importing data from Sqoop into Hive External Table with Avro encoding i had details on how you can import a table from RDBMS into Hive using Sqoop in Avro format. In that blog i went through few steps to get the avsc file, but i realized there is easier way to do it following these steps
  1. First execute the sqoop import command like this, make sure that you pass --outdir schema as parameters to the sqoop import command, what that does is it generates the CUSTOMER.avsc and CUSTOMER.java in the schema directory on your local machine
    
    sqoop import --connect jdbc:mysql://localhost/test 
    --username root 
    --password cloudera 
    --table CUSTOMER 
    --as-avrodatafile 
    --outdir schema
    
  2. You can verify that CUSTOMER.avsc file got created as you expected by executing ls -ltrA schema
  3. Next create schema directory in HDFS by executing hdfs mkdir command like this
    
    hdfs dfs -mkdir /user/cloudera/schema
    
  4. Copy the CUSTOMER.avsc from your local schema directory to HDFS in schema directory by executing following command
    
    hdfs dfs -copyFromLocal schema/CUSTOMER.avsc /user/cloudera/schema/.
    
  5. Last step is to create Hive table with CUSTOMER.avsc as schema using following command
    
    CREATE EXTERNAL TABLE CUSTOMER
    ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
    STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
    LOCATION '/user/cloudera/CUSTOMER'
    TBLPROPERTIES ('avro.schema.url'='/user/cloudera/schema/CUSTOMER.avsc');
    
Now if you go to hive and execute "SELECT * FROM CUSTOMER;" query then you should see 1 record in it like this

10 comments:

Ananthi A said...



nice blog and thanks for sharing this information...this is very useful to me...
Mobile App Development Company
Android app Development Company
ios app development Company
Mobile App Development Companies

john wick said...

Microsoft office has a different version, and the entire version has the different setup file. Some of the version is Office 2013, Office 2016 and Office 365 etc. Go to www.office.com/setup and redeem your key. Office Com Setup is one of the most popular productivity programs around. If you have your product key handy, you don't even need a disc to start installing Office just visit office setup help. Norton is one of the most reliable antivirus providers in the market to download visit www.norton.com/setup . There are multiple manufacturers making printers for home and office use with varied range as printer series per the need and affordability of different customers. Printer Tech Support. It has become one of the most useful computer hardware devices that allow people to convert computer generated typescript into a physical printed document.

sangi yadav said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in APACHE SPARK , kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training On APACHE SPARK . We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Saurabh Srivastava
MaxMunus
E-mail: saurabh@maxmunus.com
Skype id: saurabhmaxmunus
Ph:+91 8553576305 / 080 - 41103383
http://www.maxmunus.com/


sunitha vishnu said...

Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
Android Training in Chennai
Ios Training in Chennai

Install Office said...

We are providing help and support for Microsoft office Setup and activation. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get the MS Office application suite and as per your need and see how it is easy to work with Microsoft Office.

Office.com setup
www office com setup
Install Office

mcafee.com activate said...

McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.

mcafee.com activate
mcafee com activate
mcafee activate

office.com setup said...

MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.

Anonymous said...

setup.office.com

Before you plan to install the Office 2016 or Office 365 on your device be it a Computer, Laptop, Mobile Phone or a Tablet, you are required to take few important steps on of them is to remove any existing Office installations from your PC. Just like the previous Office products, Office 2016 & 365 will conflict with the previously installed versions. So, it becomes necessary to remove the previous office files properly.



setup.office.com
www.office.com/setup
office.com

www.office.com/myaccount said...


www.office.com/myaccount

To Setup retail card please visit official website Www.Office.Com/Setup. Office Retail Cards allow you to download your security product from the internet instead of installing from a CD, ensuring recent versions.



www.office.com/myaccount
www.office.com/setup
Microsoft Office product

office com setup said...

How you install or reinstall Office 365 or Office 2016 depends on whether your Office product is part of an Office for home or Office for business plan. If you're not sure what you have, see what office.com setup products are included in each plan and then follow the steps for your product. The steps below also apply if you're installing a single, stand-alone Office application such as Access 2016 or Visio 2016. Need Help with office.com/ setup Enter Product Key?
office.com set up
office com setup
microsoft office product