WordCount(HelloWorld) MapReduce program

I am learning about MapReduce and in order to experiment with MapReduce, i created this simple program which takes a text file as input and then generate a output that prints how frequently a word appeared in the text file. You can download the source code for the program from here

1. First i started by creating a simple Mapper which receives the content of the text file one line at a time, the Mapper takes care of splitting the content into words and then it writes every word into output and sets frequency count for that word to 1, by calling context.write(word,one). In this case the word becomes key and count becomes value
2. Next i had to develop a Reducer class which, receives word as key and value is list of all the counts for example if your input file is simple text like aaa bbb ccc aaa, then reduce class will get called with aaa - [1, 1], bbb -[1] and ccc - [1] as input. Hadoop framework takes care of collecting output of Mapper and then converting it into key -[value,value] format. In the reducer only thing that i had to do was to iterate through all the values and come up with a count. Once i have that i write it as output of Reducer by calling context.write(key, new IntWritable(sum));
3. The last part is creating WordCountDriver.java, which is a Java program that sets up Hadoop Framework by setting up inputs, defining outputs and also specifying name of the Mapper and Reducer class. After initializing Hadoop it calls job.waitForCompletion(true), this method will take care of passing the control to Hadoop framework and wait for the job to complete
4. Now you can either use one of the existing .txt file on your machine or you can create a text file like this

   
   hhh eee  iii bbb ccc fff ddd  ggg aaa aaa
   XXX YYY ZZZ
   hhh eee  iii bbb ccc fff ddd  ggg aaa aaa 
   XXX YYY ZZZ
   hhh eee  iii bbb ccc fff ddd  ggg aaa aaa 
   XXX YYY ZZZ
   hhh eee  iii bbb ccc fff ddd  ggg aaa aaa 
   hhh eee  iii bbb ccc fff ddd  ggg aaa aaa
   

5. Last step is to run your Hadoop program, if you used the Eclipse or some other IDE for developing your code, you can run your program directly by running WordCountDriver.java directly. This program takes 2 parameters, in my case since the input file is on local file system and i want the output to get stored on local file system too, i pass following 2 parameters

   
   file:///Users/sunil/hadoop/sorttest.txt file:///Users/sunil/hadoop/output/wordcount
   

6. Once the program is finished successfully, you would be able to see part-r-00000 file created on your local machine at /Users/sunil/hadoop/output/wordcount, if you open it you should see output like this

   
   XXX     3
   YYY     3
   ZZZ     3
   aaa     10
   bbb     5
   ccc     5
   ddd     5
   eee     5
   fff     5
   ggg     5
   hhh     5
   iii     5
   

If you want to run this program with bigger text file then you can download few good classical books from Algorithm site data section