The MapReduce framework passes control to your combiner class at the end of the map phase to combine different output files generated by Mappers, so that your combiner class combines/reduce the data generated by Mappers before it gets transferred to the Reducers. Sending data from Mapper to reducer requires that data to go over network from Mapper to Reducer.
I wanted to try creating custom combiner class, In order to keep things simple i decided to add combiner class in
WordCount(HelloWorld) MapReduce program . Basically my combiner class does same thing as reducer, which is to take multiple [word, 1] tuples and combine them into something like [word1, 5], [word2, 6],,, etc. I followed these steps
-
First thing that i did was to create WordCountCombiner.java class that looks same as that of
WordCountReducer, but i did add one System.Out.println() in it so that i would know when my combiner is called instead of reducer.
-
Then i changed the driver class for my MapReduce framework class to add
job.setCombinerClass(WordCountCombiner.class); line in it.
Then i did execute the WordCountDriver class with 3 files as input and i could see my Combiner class getting called after the Mapper class for each input file, before it wrote the mapper output to the disk and before starting reducer phase.
