Hadoop MapReduce HTTP Notification

Normally MapReduce programs tend to run for long time and you might want to setup a way to get notification when the job is done finishing to find out if the job was executed successfully or not. Hadoop provides a mechanism in which you can get notification on the progress of your MapReduce job. This is two step process
  1. First set up job.end.notification.url property with value equal to a web application URL that should get invoked with the progress of job job.getConfiguration().set("job.end.notification.url", "http://localhost:8080/mrnotification/MRNotification/$jobId?status=$jobStatus");
  2. Next create a web application that will receive the notification and printout the job status and job name like this

1 comment:

Akshay said...

I am trying to work on getting a notification once the job is completed. I have Implemented a listener like this.

import java.io.*;
import java.util.Enumeration;
import javax.servlet.*;
import javax.servlet.http.*;
import java.sql.*;

public class Serv extends HttpServlet
{
private static final long serialVersionUID = 1L;
public void doGet(HttpServletRequest req,HttpServletResponse res)throws ServletException,IOException
{
String jobId=req.getParameter("jobId");
String jobStatus=req.getParameter("jobStatus");
res.setContentType("text/html");
PrintWriter pw=res.getWriter();
pw.println("JobId: \n"+jobId);
pw.println("JobStatus: \n"+jobStatus);
pw.close();
}
public void doPost(HttpServletRequest req,HttpServletResponse res)throws ServletException,IOException
{
String jobId=req.getParameter("jobId");
String jobStatus=req.getParameter("jobStatus");
res.setContentType("text/html");
PrintWriter pw=res.getWriter();
pw.println("JobId: \n"+jobId);
pw.println("JobStatus: \n"+jobStatus);
pw.close();
}
}

This is the part of code for notification in my Toolrunner run function.

conf.set("job.end.notification.url", "http://localhost:8080/Serv?jobId=$jobId&jobStatus=$jobStatus");
conf.setInt("job.end.retry.attempts", 3);
conf.setInt("job.end.retry.interval", 1000);

What I get after this is just a Null Null string with the url.
This is my question posted on Stack Overflow. http://stackoverflow.com/questions/24857524/implementing-job-end-notification-listener-in-hadoop
Any help is appreciated.