Using Apache Oozie to execute MapReduce jobs

I wanted to learn about how to automate MapReduce job using Oozie, so i decide to create Oozie workflow to invoke WordCount(HelloWorld) MapReduce program. I had to follow these steps
  1. FIrst thing that i did was to download the WordCount program source code by executing
    
    git clone https://github.com/sdpatil/HadoopWordCount3
    
    This program does have maven script for building executable jar, so i used mvn clean package command to build Hadoop jar.
  2. After that i tried executing the program manually by using following following command
    
    hadoop jar target/HadoopWordCount.jar sorttest.txt output/wordcount
    
  3. Now in order to use Oozie workflow you will have to create a particular folder structure on your machine
    
    wordcount
       -- job.properties
       -- workflow.xml
       -- lib
             -- HadoopWordCount.jar  
    
  4. In the workcount folder create job.properties file like this, This file lets you pass parameters to your oozie workflow. Value of nameNode and jobTracker represent the name node and job tracker location. In my case i am using cloudera vm with single ndoe so both these properties point to localhost. The value of oozie.wf.application.path is equal to HDFS path where you uploaded the wordcount folder created in step 3
  5. Next define your Apache oozie workflow.xml file like this. In my case the workflow has single step which is to execute mapreduce job. I am
    • mapred.mapper.new-api & mapred.reducer.new-api: Set this property to true if your using the new MapReduce API based on org.apache.hadoop.mapreduce.* classes
    • mapreduce.map.class: The fully qualified name of your mapper class
    • mapreduce.reduce.class: The fully qualified name of your reducer class
    • mapred.output.key.class: Fully qualified name of the output key class. This is same as parameter to job.setOutputKeyClass() in your driver class
    • mapred.output.value.class: Fully qualified name of the output value class. This is same as parameter to job.setOutputValueClass() in your driver class
    • mapred.input.dir: Location of your input file in my case i have sorttext.txt in hdfs://localhost/user/cloudera directory
    • mapred.output.dir:Location of output file that will get generated. In my case i want output to go to hdfs://localhost/user/cloudera/output/wordcount directory
  6. Once your oozie workflow is ready upload the wordcount folder in HDFS by executing following command
    
    hdfs dfs -put oozie wordcount
    
  7. 
    Now run your oozie workflow by executing following command from your wordcount directory
    oozie job -oozie http://localhost:11000/oozie -config job.properties -run
    
    If it runs successfully you should see output generated in hdfs://localhost/user/cloudera/output/wordcount directory

20 comments:

Subu said...

Cool summary!

srjwebsolutions said...

We are leading responsive website designing and development company in Noida.
We are offering mobile friendly responsive website designing, website development, e-commerce website, seo service and sem services in Noida.

Responsive Website Designing Company in Noida
Website Designing Company in Noida
SEO Services in Noida
SMO Services in Noida

BatteryMantra No. 1online battery store said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

BatteryMantra No. 1online battery store said...

Battery Mantra is Authorized exide car battery dealer in Noida and Greater Noida. We are providing our service in Indirapuram, Delhi, Ashok Nagar.

Exide Battery Dealer in Noida
Battery Dealer in Noida
Authorized Battery Dealer in Noida
Car Battery Dealer in Noida
Car Battery Dealer
Exide Battery Dealer

Genxsoft said...

Egmedi.com is online medical store pharmacy in laxmi nagar Delhi. You can Order prescription/OTC medicines online. Cash on Delivery available. Free Home Delivery


Online Pharmacy in Delhi
Buy Online medicine in Delhi
Online Pharmacy in laxmi nagar
Buy Online medicine in laxmi nagar
Onine Medical Store in Delhi
Online Medical store in laxmi nagar
Online medicine store in delhi
online medicine store in laxmi nagar
Purchase Medicine Online
Online Pharmacy India
Online Medical Store

Tejuteju said...

It was the very nice article and it is very useful Big data Hadoop online training

sheela rajesh said...

This blog is more effective and it is very much useful for me.we need more information please keep update more.
Hadoop Training in Chennai
Big data training in chennai
big data training in velachery
JAVA Training in Chennai
Python Training in Chennai
Software testing training in chennai
hadoop training in Annanagar
big data training in chennai anna nagar
Big data training in annanagar

Abhi said...

Thanks for info....
Website development in Bangalore

Abhi said...


Thanks for info
Website development in Bangalore

Unknown said...

Thank you for this blog. I was about to sign up to first rate when this showed up in the results. I hope I can find something that can really help me out. I do not drive so an online work will be great.It really makes me happy and I am satisfied with the arrangement of your post. You are really a talented person I have ever seen.Geek Squad Tech Support visit for more information

Inoventic Creative Agency said...

Hello Admin!

Thanks for the post. It was very interesting and meaningful. I really appreciate it! Keep updating stuffs like this. If you are looking for the Advertising Agency in Chennai | Printing in Chennai , Visit Inoventic Creative Agency Today..

Rajesh Verma said...

Much obliged for your article. It was intriguing and useful.

Here I additionally need to recommend your peruser who normally heads out from one spot to another they should visit Airlines Gethuman that offer the best arrangements to book your seat on Delta Airlines Reservations. Hurry do as well and benefits the best arrangements and dispose of to check various sites for offers.
Get more help on
Southwest Airlines Flights
Delta Airlines Ticketing

Southwest Airlines Reservations said...

Thanks for your article. It was interesting and informative.
Here I also want to suggest your reader who usually travels one place to another they must visit Airlines Gethuman that offer best deals to book your seat on Delta Airlines Reservations. So do hurry and avail the best deals and get rid of to check different websites for offers

Southwest Airlines Reservations

Southwest Airlines Reservations said...

Airlines-gethuman.org is a platform where you can Find Contact Information for all the airlines, Assistance with booking your flights, and vacation packages easily. It helps you to save both, your money and your time.


Southwest Airlines Reservations

Southwest Airlines Flights

Lottery sambad said...

To see who is going to become a millionaire from the Lottery Sambad result today, download today's Lottery Sambad result on this portal.After the bygone era, playing a lottery game is now an easy way to earn money in an exciting way.The lottery sambad Morning West Bengal State Lottery Nagaland state Lottery Kerala Lottery Result Kerala Lottery Today Result published three times a day, is a very quality lottery and Satta king results.

Yakshita said...

Thanks for Sharing such an amazing article. Keep working... Your Site is very nice, and it's very helping us.. this post is unique and interesting, thank you for sharing this awesome information

IDFC First Bank Netbanking Registration 2021 | IDFC First Bank Netbanking Login 2021

Zoom said...

Really loved this website content and design, visit Niagara Cab Company
niagara falls cab company

realtorsophia said...

Thanks for your blog. I also want to share some information with those who are looking for a new home in California, then please contact us. we are an innovative realtor of housesofpayne to help you to sell and buy homes, find the value of your home for real estate project anywhere in California or ca.
Visit for more information on:
Realtors in Nicasio ca
Home Values in Petaluma ca

Bipard said...

I hope to see more post from you. Thank you for sharing this post. Your blog posts are more interesting and impressive

TN Elections Portal, TN Voter Id Registration, Apply Online, Status and List Check, elections.tn.gov.in

James Alter said...

What makes Delta Airlines so great is the fact that the airline provides great customer service over the phone and online. You will find Delta Airlines Office in Kampala. Delta has deployed a team of experts in its office that has answers to all the questions and queries of customers.