Importing data from Sqoop into Hive External Table with Avro encoding updated

In the Importing data from Sqoop into Hive External Table with Avro encoding i had details on how you can import a table from RDBMS into Hive using Sqoop in Avro format. In that blog i went through few steps to get the avsc file, but i realized there is easier way to do it following these steps
  1. First execute the sqoop import command like this, make sure that you pass --outdir schema as parameters to the sqoop import command, what that does is it generates the CUSTOMER.avsc and CUSTOMER.java in the schema directory on your local machine
    
    sqoop import --connect jdbc:mysql://localhost/test 
    --username root 
    --password cloudera 
    --table CUSTOMER 
    --as-avrodatafile 
    --outdir schema
    
  2. You can verify that CUSTOMER.avsc file got created as you expected by executing ls -ltrA schema
  3. Next create schema directory in HDFS by executing hdfs mkdir command like this
    
    hdfs dfs -mkdir /user/cloudera/schema
    
  4. Copy the CUSTOMER.avsc from your local schema directory to HDFS in schema directory by executing following command
    
    hdfs dfs -copyFromLocal schema/CUSTOMER.avsc /user/cloudera/schema/.
    
  5. Last step is to create Hive table with CUSTOMER.avsc as schema using following command
    
    CREATE EXTERNAL TABLE CUSTOMER
    ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
    STORED AS INPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
    OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
    LOCATION '/user/cloudera/CUSTOMER'
    TBLPROPERTIES ('avro.schema.url'='/user/cloudera/schema/CUSTOMER.avsc');
    
Now if you go to hive and execute "SELECT * FROM CUSTOMER;" query then you should see 1 record in it like this

27 comments:

  1. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in APACHE SPARK , kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training On APACHE SPARK . We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.
    For Demo Contact us.
    Saurabh Srivastava
    MaxMunus
    E-mail: saurabh@maxmunus.com
    Skype id: saurabhmaxmunus
    Ph:+91 8553576305 / 080 - 41103383
    http://www.maxmunus.com/


    ReplyDelete
  2. McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.

    mcafee.com activate
    mcafee com activate
    mcafee activate

    ReplyDelete
  3. MS Office setup is very easy to install, download and redeem. Use of MS Office is also simple and the user can learn the use of it easily. Online help option is also available in all application of the MS Office which provides an instant guideline.

    ReplyDelete
  4. norton.com/setupOnline Help – Step by Step guide for Norton Setup, Download & complete installation online. We are providing independent support service if in case you face problem to activate or Setup Norton product.

    ReplyDelete
  5. Mcafee Total Protection provides an all-rounder protection to all your devices against many threats like virus, malware, spyware, trojan, Ransomware and much more. Furthermore, you do not need additional products for your PC, Laptop, Tablet, and Smartphones. Also, McAfee provides free support via calls, chat or mail services to their customer and guide them through any McAfee product issue. This is an ideal product for users to maintain a secure digital life on every device all the time.mcafee.com/mtp/retailcard

    ReplyDelete


  6. Wow great! Being in work environment i learnt something new ,i would use it in future. Hadoop Training in Chennai

    ReplyDelete
  7. Hi Your Blog Is Very Nice! Attractive. Content is Nice

    Stumagz is ultimate platform to release student magazine articles and engineering college news. It helps students to share their ideas in form of article.

    stuMagz is an online platform that brings all the students and colleges together. Despite the fact that NBA accreditation says that every college must maintain a magazine to publish their content, there is no proper platform for students to expose themselves to the ecosystem and the lack of reach for the students to the various opportunities present in the market.There is a lack of connectivity between different colleges and universities.

    The idea of stuMagz was born to bridge this gap and expand the scope for learning. It is a simple and efficient platform which provides every college and its students, a hassle free experience to publish their content and unleash their creativity.

    Student Magazine Articles

    Digital Campus Eco-System

    Digital Stories In Hyderabad

    Digital Classrooms In Hyderabad

    Student Magazine Subscriptions

    College Magazine Articles

    College Fest Event

    Top Engineering Colleges In India

    For more Details Visit Us: Stumagz.com

    ReplyDelete
  8. Hi,
    Your article is very informative and has a lot of information. I really like your effort, keep posting.

    Quicken® Support Number for USA and Canada +1-888-586-5828. (24/7 Toll-Free)

    Any Error In Quicken Software Dial Quicken Support Number 1-888-586-5828 Get Instant Help and Support from Quicken Experts Customer Support Service Center 24/7.

    For More Help Call Our Toll free:- (+1)-888-586-5828.
    Website:- http://intuittechnicalsupports.com/

    ReplyDelete
  9. As we all know that printers are used by more than millions of users across the globe. But some time users are facing different type of problem in their Printer, if you are one of them user and you want to solve your printer issue then you need to visit here: HP Printer Support | Epson Printer Support | Lexmark Printer Support | Brother Printer Support | Canon Printer Support

    ReplyDelete
  10. HIPAA Privacy Officer Training will help to understand safeguards for keeping protected health information safe from a people, administrative, and contractual standpoint

    ReplyDelete
  11. This is the most important step toward compliance. If your HIPAA officer does not know what is needed to achieve and maintain compliance, the company will not be compliant. This 25 hours training of Certified HIPAA Security Officer Expert Training(CHPSE) will give them a complete knowledge of what are regulations requirements.

    ReplyDelete
  12. We offer complete help for your Primevideo.com/mytv device. From the underlying purpose of initiating your gadget till where your channels are flawlessly spilling, we provide step by step assistance. You may have inquiries on initiating the gadget and setting up with all the most loved channels and putting it up inside and out; Prime video helpline has all the orders to sift through all the inquiry you have. We assist with fixing your myTV gadgets, be it initiating it or becoming acquainted with about the Channel list on Roku or in any event, repairing your Roku remote. If you facing any queries related to prime video or mytv, Roku or Android devices Setup contact us, our expert's team always available customer help.

    ReplyDelete
  13. Thanks for sharing this great post. If you need any kind of Amazon myTV activate related any queries contact us. Amazon prime customer care team always available for customer suitability and fix your issue instantly.

    ReplyDelete
  14. It's an interesting and informative article..!! Thanks for sharing. To know how to set up and fix a brother printer issues, Check in to our site..!!
    Brother Printer Setup | Brother Printer Installation | Brother Printer Driver | Brother Printer Wireless | Brother Printer Printing | Brother Printer Troubleshooting

    ReplyDelete
  15. I absolutely love your blog.. Pleasant colors & theme.

    Did you develop this website yourself? Please reply back as I’m wanting to create my
    own personal website and would like to find out where you got
    this from or exactly what the theme is called.
    Thanks!

    ReplyDelete
  16. It's excellent. You will certainly really feel that your writing is extraordinary even if you take a look at it. I enjoyed to see your writing and also my sensations are still sticking around. I wish you will certainly be impressed similar to me when you see my writing. Would certainly you such as to find and also see my message? 바카라사이트

    ReplyDelete
  17. Look For Prime video Com Mytv Activation Code. Get Results from Several Engines at Once. Let fast quick search Find Your Results Today. Here we have everything you need. Information. Data. Replies. Information. Results. Arrangements. Administrations: Find What You Look For, Useful Information.

    Read more...
    primevideo.com/mytv code
    primevideo/mytv code

    ReplyDelete