Importing data from Sqoop into Hive External Table with Avro encoding updated

In the Importing data from Sqoop into Hive External Table with Avro encoding i had details on how you can import a table from RDBMS into Hive using Sqoop in Avro format. In that blog i went through few steps to get the avsc file, but i realized there is easier way to do it following these steps
  1. First execute the sqoop import command like this, make sure that you pass --outdir schema as parameters to the sqoop import command, what that does is it generates the CUSTOMER.avsc and CUSTOMER.java in the schema directory on your local machine
    
    sqoop import --connect jdbc:mysql://localhost/test 
    --username root 
    --password cloudera 
    --table CUSTOMER 
    --as-avrodatafile 
    --outdir schema
    
  2. You can verify that CUSTOMER.avsc file got created as you expected by executing ls -ltrA schema
  3. Next create schema directory in HDFS by executing hdfs mkdir command like this
    
    hdfs dfs -mkdir /user/cloudera/schema
    
  4. Copy the CUSTOMER.avsc from your local schema directory to HDFS in schema directory by executing following command
    
    hdfs dfs -copyFromLocal schema/CUSTOMER.avsc /user/cloudera/schema/.
    
  5. Last step is to create Hive table with CUSTOMER.avsc as schema using following command
    
    CREATE EXTERNAL TABLE CUSTOMER
    ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
    STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
    LOCATION '/user/cloudera/CUSTOMER'
    TBLPROPERTIES ('avro.schema.url'='/user/cloudera/schema/CUSTOMER.avsc');
    
Now if you go to hive and execute "SELECT * FROM CUSTOMER;" query then you should see 1 record in it like this

32 comments:

Ananthi A said...



nice blog and thanks for sharing this information...this is very useful to me...
Mobile App Development Company
Android app Development Company
ios app development Company
Mobile App Development Companies

john wick said...

Microsoft office has a different version, and the entire version has the different setup file. Some of the version is Office 2013, Office 2016 and Office 365 etc. Go to www.office.com/setup and redeem your key. Office Com Setup is one of the most popular productivity programs around. If you have your product key handy, you don't even need a disc to start installing Office just visit office setup help. Norton is one of the most reliable antivirus providers in the market to download visit www.norton.com/setup . There are multiple manufacturers making printers for home and office use with varied range as printer series per the need and affordability of different customers. Printer Tech Support. It has become one of the most useful computer hardware devices that allow people to convert computer generated typescript into a physical printed document.

sangi yadav said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in APACHE SPARK , kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training On APACHE SPARK . We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Saurabh Srivastava
MaxMunus
E-mail: saurabh@maxmunus.com
Skype id: saurabhmaxmunus
Ph:+91 8553576305 / 080 - 41103383
http://www.maxmunus.com/


sunitha vishnu said...

Really it was an awesome article...very interesting to read..You have provided an nice article....Thanks for sharing..
Android Training in Chennai
Ios Training in Chennai

Install Office said...

We are providing help and support for Microsoft office Setup and activation. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get the MS Office application suite and as per your need and see how it is easy to work with Microsoft Office.

Office.com setup
www office com setup
Install Office

mcafee.com activate said...

McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.

mcafee.com activate
mcafee com activate
mcafee activate

office.com setup said...

MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.

Anonymous said...

setup.office.com

Before you plan to install the Office 2016 or Office 365 on your device be it a Computer, Laptop, Mobile Phone or a Tablet, you are required to take few important steps on of them is to remove any existing Office installations from your PC. Just like the previous Office products, Office 2016 & 365 will conflict with the previously installed versions. So, it becomes necessary to remove the previous office files properly.



setup.office.com
www.office.com/setup
office.com

www.office.com/myaccount said...


www.office.com/myaccount

To Setup retail card please visit official website Www.Office.Com/Setup. Office Retail Cards allow you to download your security product from the internet instead of installing from a CD, ensuring recent versions.



www.office.com/myaccount
www.office.com/setup
Microsoft Office product

office com setup said...

How you install or reinstall Office 365 or Office 2016 depends on whether your Office product is part of an Office for home or Office for business plan. If you're not sure what you have, see what office.com setup products are included in each plan and then follow the steps for your product. The steps below also apply if you're installing a single, stand-alone Office application such as Access 2016 or Visio 2016. Need Help with office.com/ setup Enter Product Key?
office.com set up
office com setup
microsoft office product

setup.office.com said...

setup.office.com

Before you plan to install the Office 2016 or Office 365 on your device be it a Computer, Laptop, Mobile Phone or a Tablet, you are required to take few important steps on of them is to remove any existing Office installations from your PC. Just like the previous Office products, Office 2016 & 365 will conflict with the previously installed versions. So, it becomes necessary to remove the previous office files properly.


setup.office.com
www.office.com/setup
office.com

www.office.com/myaccount said...

www.office.com/myaccount

To Setup retail card please visit official website Www.Office.Com/Setup. Office Retail Cards allow you to download your security product from the internet instead of installing from a CD, ensuring recent versions.


www.office.com/myaccount
www.office.com/setup
Microsoft Office product

Smith said...

norton.com/setupOnline Help – Step by Step guide for Norton Setup, Download & complete installation online. We are providing independent support service if in case you face problem to activate or Setup Norton product.

Smith said...

Mcafee Total Protection provides an all-rounder protection to all your devices against many threats like virus, malware, spyware, trojan, Ransomware and much more. Furthermore, you do not need additional products for your PC, Laptop, Tablet, and Smartphones. Also, McAfee provides free support via calls, chat or mail services to their customer and guide them through any McAfee product issue. This is an ideal product for users to maintain a secure digital life on every device all the time.mcafee.com/mtp/retailcard

Smith said...

Office.com/verify – To download and install the Microsoft Office set up on your system it is necessary to have a verified Microsoft account. Microsoft Word offers you the feature of previewing the document and navigation pane. Microsoft PowerPoint helps you to prepare the professional presentation for office work or school work. The Microsoft Office is a complete tool which makes the tasks easier and also optimize the system for better performance.

Smith said...

Online Supports...
TurboTax Support
Quicken Support
Quickbooks Support

quicken update said...

awesome post man ! thanks for this useful blog .. looking for .www.office.com/setup..then just feel free to call us on 1-844-666-8616

Aman tripathi said...

Webroot Support - Call 1-844-456-8733 (US/CA) for Webroot Installation, Activation, Configuration, Billing, Renewal or any other queries. Contact our Webroot technical support to get Best Webroot Customer Service\
webroot support

Aman tripathi said...

dell 1-844-456-8733 (US/CA) any Installation, Drivers, Repair, Errors and many more. Get Online Resolution anytime by Technical Support.
dell printer support

Aman tripathi said...

Epson Printer Support - Get Quick Setup for Wireless Epson Printer Install by calling
Epson Printer Support number 1-844-456-8733 (US/CA) Toll-free.
Epson Printer Support

Aman tripathi said...

HP Printer Support - Call HP Technical Support Phone Number 1-844-456-8733 (US/CA) or 01444 39 0119 (UK) for any HP Printer related issues.
hp Printer Support

Aman tripathi said...

Garmin Download at www.garmin.com/express. Register, Update and sync your Garmin Express today and get started with your Garmin Maps.
garmin.com/express

Aman tripathi said...

Brother Printer Support - - Brother Printer Support 1-844-456-8733 (US/CA) any Installation, Drivers, Repair, Errors and many more. Get Online Resolution anytime by Technical Support.
Brother Printer Support

john wick said...

Microsoft office setup is the software setup file with this setup file you can install on your computer and some of the supported device to use Microsoft office.
www.office.com/setup
office.com/setup
office com setup

sowmiya gopal said...



Wow great! Being in work environment i learnt something new ,i would use it in future. Hadoop Training in Chennai

Aman tripathi said...

Netgear Router Support
Belkin Router Support
D-Link Router Support
Cisco Router Support
McAfee.com/Activate
Roku.com/link

Peter Berg said...

office.com/setup
office.com/setup
www.office.com/setup

Stumagz.com said...

Hi Your Blog Is Very Nice! Attractive. Content is Nice

Stumagz is ultimate platform to release student magazine articles and engineering college news. It helps students to share their ideas in form of article.

stuMagz is an online platform that brings all the students and colleges together. Despite the fact that NBA accreditation says that every college must maintain a magazine to publish their content, there is no proper platform for students to expose themselves to the ecosystem and the lack of reach for the students to the various opportunities present in the market.There is a lack of connectivity between different colleges and universities.

The idea of stuMagz was born to bridge this gap and expand the scope for learning. It is a simple and efficient platform which provides every college and its students, a hassle free experience to publish their content and unleash their creativity.

Student Magazine Articles

Digital Campus Eco-System

Digital Stories In Hyderabad

Digital Classrooms In Hyderabad

Student Magazine Subscriptions

College Magazine Articles

College Fest Event

Top Engineering Colleges In India

For more Details Visit Us: Stumagz.com

srjwebsolutions said...


We are leading responsive website designing and development company in Noida.
We are offering mobile friendly responsive website designing, website development, e-commerce website, seo service and sem services in Noida.

Responsive Website Designing Company in Noida
Website Designing Company in Noida
SEO Services in Noida
SMO Services in Noida

Vikas Chaudhary said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

EG MEDI said...

Egmedi.com is online medical store pharmacy in laxmi nagar Delhi. You can Order prescription/OTC medicines online. Cash on Delivery available. Free Home Delivery


Online Pharmacy in Delhi
Buy Online medicine in Delhi
Online Pharmacy in laxmi nagar
Buy Online medicine in laxmi nagar
Onine Medical Store in Delhi
Online Medical store in laxmi nagar
Online medicine store in delhi
online medicine store in laxmi nagar
Purchase Medicine Online
Online Pharmacy India
Online Medical Store

Teju Teju said...

awesome post presented by you..your writing style is fabulous and keep updated with your blogs Big data Hadoop online Course Bangalore
Big data Hadoop online training
Big data hadoop online training India