Importing data from Sqoop into Hive External Table with Avro encoding updated

In the Importing data from Sqoop into Hive External Table with Avro encoding i had details on how you can import a table from RDBMS into Hive using Sqoop in Avro format. In that blog i went through few steps to get the avsc file, but i realized there is easier way to do it following these steps
  1. First execute the sqoop import command like this, make sure that you pass --outdir schema as parameters to the sqoop import command, what that does is it generates the CUSTOMER.avsc and CUSTOMER.java in the schema directory on your local machine
    
    sqoop import --connect jdbc:mysql://localhost/test 
    --username root 
    --password cloudera 
    --table CUSTOMER 
    --as-avrodatafile 
    --outdir schema
    
  2. You can verify that CUSTOMER.avsc file got created as you expected by executing ls -ltrA schema
  3. Next create schema directory in HDFS by executing hdfs mkdir command like this
    
    hdfs dfs -mkdir /user/cloudera/schema
    
  4. Copy the CUSTOMER.avsc from your local schema directory to HDFS in schema directory by executing following command
    
    hdfs dfs -copyFromLocal schema/CUSTOMER.avsc /user/cloudera/schema/.
    
  5. Last step is to create Hive table with CUSTOMER.avsc as schema using following command
    
    CREATE EXTERNAL TABLE CUSTOMER
    ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
    STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
    LOCATION '/user/cloudera/CUSTOMER'
    TBLPROPERTIES ('avro.schema.url'='/user/cloudera/schema/CUSTOMER.avsc');
    
Now if you go to hive and execute "SELECT * FROM CUSTOMER;" query then you should see 1 record in it like this

27 comments:

Unknown said...

I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in APACHE SPARK , kindly contact us http://www.maxmunus.com/contact
MaxMunus Offer World Class Virtual Instructor led training On APACHE SPARK . We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
For Demo Contact us.
Saurabh Srivastava
MaxMunus
E-mail: saurabh@maxmunus.com
Skype id: saurabhmaxmunus
Ph:+91 8553576305 / 080 - 41103383
http://www.maxmunus.com/


mcafee.com activate said...

McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.

mcafee.com activate
mcafee com activate
mcafee activate

office.com setup said...

MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.

Smith said...

norton.com/setupOnline Help – Step by Step guide for Norton Setup, Download & complete installation online. We are providing independent support service if in case you face problem to activate or Setup Norton product.

Smith said...

Mcafee Total Protection provides an all-rounder protection to all your devices against many threats like virus, malware, spyware, trojan, Ransomware and much more. Furthermore, you do not need additional products for your PC, Laptop, Tablet, and Smartphones. Also, McAfee provides free support via calls, chat or mail services to their customer and guide them through any McAfee product issue. This is an ideal product for users to maintain a secure digital life on every device all the time.mcafee.com/mtp/retailcard

Unknown said...



Wow great! Being in work environment i learnt something new ,i would use it in future. Hadoop Training in Chennai

peter berg said...

office.com/setup
office.com/setup
www.office.com/setup

Stumagz.com said...

Hi Your Blog Is Very Nice! Attractive. Content is Nice

Stumagz is ultimate platform to release student magazine articles and engineering college news. It helps students to share their ideas in form of article.

stuMagz is an online platform that brings all the students and colleges together. Despite the fact that NBA accreditation says that every college must maintain a magazine to publish their content, there is no proper platform for students to expose themselves to the ecosystem and the lack of reach for the students to the various opportunities present in the market.There is a lack of connectivity between different colleges and universities.

The idea of stuMagz was born to bridge this gap and expand the scope for learning. It is a simple and efficient platform which provides every college and its students, a hassle free experience to publish their content and unleash their creativity.

Student Magazine Articles

Digital Campus Eco-System

Digital Stories In Hyderabad

Digital Classrooms In Hyderabad

Student Magazine Subscriptions

College Magazine Articles

College Fest Event

Top Engineering Colleges In India

For more Details Visit Us: Stumagz.com

Tejuteju said...

awesome post presented by you..your writing style is fabulous and keep updated with your blogs Big data Hadoop online Course Bangalore
Big data Hadoop online training
Big data hadoop online training India


jolly said...

thanks!!
Avast Support Number

jolly said...

best one!!
mcafee.com/activate

jolly said...

wow!! fabulous blog!!
mcafee.com/activate

Raj Kumar Singh said...

Hi,
Your article is very informative and has a lot of information. I really like your effort, keep posting.

Quicken® Support Number for USA and Canada +1-888-586-5828. (24/7 Toll-Free)

Any Error In Quicken Software Dial Quicken Support Number 1-888-586-5828 Get Instant Help and Support from Quicken Experts Customer Support Service Center 24/7.

For More Help Call Our Toll free:- (+1)-888-586-5828.
Website:- http://intuittechnicalsupports.com/

Unknown said...

refreshing poost!!
www.office.com/setup

Anbarasan14 said...

Great post. You have written a valuable content in a interesting way. Kindly share more updates.

IELTS Classes in T Nagar
IELTS Coaching In Velachery
IELTS Coaching in Tambaram
Spoken English Classes in Chennai
Best Spoken English Classes in Chennai
English Speaking Classes in Mumbai
IELTS Coaching in Chennai
IELTS Classes in Mumbai

jenifferleio12 said...

As we all know that printers are used by more than millions of users across the globe. But some time users are facing different type of problem in their Printer, if you are one of them user and you want to solve your printer issue then you need to visit here: HP Printer Support | Epson Printer Support | Lexmark Printer Support | Brother Printer Support | Canon Printer Support

Abhi said...

Thanks for info....
Website development in Bangalore

Christoper stalin said...

It is very useful blog!!!! Thanks for sharing this blog post.
web development classes | web designing and development course
php developer course | php course in chennai
magento course in chennai | magento developer training

sahil sharma said...

HIPAA Privacy Officer Training will help to understand safeguards for keeping protected health information safe from a people, administrative, and contractual standpoint

sahil sharma said...

This is the most important step toward compliance. If your HIPAA officer does not know what is needed to achieve and maintain compliance, the company will not be compliant. This 25 hours training of Certified HIPAA Security Officer Expert Training(CHPSE) will give them a complete knowledge of what are regulations requirements.

primevideocommytv said...

We offer complete help for your Primevideo.com/mytv device. From the underlying purpose of initiating your gadget till where your channels are flawlessly spilling, we provide step by step assistance. You may have inquiries on initiating the gadget and setting up with all the most loved channels and putting it up inside and out; Prime video helpline has all the orders to sift through all the inquiry you have. We assist with fixing your myTV gadgets, be it initiating it or becoming acquainted with about the Channel list on Roku or in any event, repairing your Roku remote. If you facing any queries related to prime video or mytv, Roku or Android devices Setup contact us, our expert's team always available customer help.

James Mill said...

Thanks for sharing this great post. If you need any kind of Amazon myTV activate related any queries contact us. Amazon prime customer care team always available for customer suitability and fix your issue instantly.

Edwin Robert said...

It's an interesting and informative article..!! Thanks for sharing. To know how to set up and fix a brother printer issues, Check in to our site..!!
Brother Printer Setup | Brother Printer Installation | Brother Printer Driver | Brother Printer Wireless | Brother Printer Printing | Brother Printer Troubleshooting

Techsolve Solutions said...

Thanks for sharing useful information. Keep on sharing.

Web developer mississauga
Digital marketing agency mississauga

카지노SEO said...

I absolutely love your blog.. Pleasant colors & theme.

Did you develop this website yourself? Please reply back as I’m wanting to create my
own personal website and would like to find out where you got
this from or exactly what the theme is called.
Thanks!

unknow said...

It's excellent. You will certainly really feel that your writing is extraordinary even if you take a look at it. I enjoyed to see your writing and also my sensations are still sticking around. I wish you will certainly be impressed similar to me when you see my writing. Would certainly you such as to find and also see my message? 바카라사이트

Primevideomytv said...

Look For Prime video Com Mytv Activation Code. Get Results from Several Engines at Once. Let fast quick search Find Your Results Today. Here we have everything you need. Information. Data. Replies. Information. Results. Arrangements. Administrations: Find What You Look For, Useful Information.

Read more...
primevideo.com/mytv code
primevideo/mytv code