Setting Hang Detection policy using wsadmin script

You can set the hang detection policy using the following admin script

server = AdminControl.completeObjectName("type=Server,*")
# Print existing values of the attributes to console
print AdminControl.getAttribute(server,"threadMonitorInterval")
print AdminControl.getAttribute(server,"threadMonitorThreshold")
print AdminControl.getAttribute(server,"threadMonitorAdjustmentThreshold")

# Set new attribute values
AdminControl.setAttribute(server,"threadMonitorInterval","70")
AdminControl.setAttribute(server,"threadMonitorThreshold","70")
AdminControl.setAttribute(server,"threadMonitorAdjustmentThreshold","70")


Please note one very important point that AdminControl object is used for making changes in the runtime environment, so the changes that you make using this script would be effective immediately, and you will loose those changes if you restart the server

I could not find attribute that allows you to generate javacore on hanged thread.

Hang Detection Policy Example

I wanted to try hang detection policy so i decided to create a sample application that will first create hang and then i will use the WAS tools to debug the issue. I followed these steps

• Create a sample HangDetectionServlet as shown in the listing
  
public class HangDetectionServlet extends HttpServlet {
private static final long serialVersionUID = 1L;

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
DateFormat d = new SimpleDateFormat("mm:ss:SS");
response.getWriter().println("Entering HangDetectionServlet.doGet() " + d.format(new Date()));
try {
Thread.sleep(250000);
} catch (InterruptedException e) {
e.printStackTrace();
}
response.getWriter().println("Exiting HangDetectionServlet.doGet() " + d.format(new Date()));
}

}

  This servlet is very simple, when it gets HTTP GET request it puts the current thread in sleep for 250 seconds.
  


• By default WAS marks a thread has hanged thread if it is running for more than 10 minutes. So our HangDetectionServlet will not be marked as hanged. So in the next step we will change the hang detection policy on WAS so that if the thread is running for more than 60 seconds/ 1 minute it will be considered hanged.


• You can set Hang Detection policy on WAS using WAS Admin Console. First Login into WAS admin console and then go to Servers < Application Servers < server_name. Then under Server Infrastructure go to Administration -> Custom Properties.
  
  


• Set hang detection policies like this
  
  By setting value of com.ibm.websphere.threadmonitor.threshold to 60 i am saying that if the thread is running for more than 60 seconds then it should be considered hanged and setting com.ibm.websphere.threadmonitor.dump.java to true means when application server detects hanged thread it should generate thread dump in addition to writing message in the SysetemOut.log. Setting value of com.ibm.websphere.threadmonitor.interval to 60 means saying that the thread monitor should run every 60 seconds to check for hanged thread. After setting these values restart the server for changes to take effect


• Now deploy the HangDetectionServlet on your server and access it, after couple of minutes i could see this message in the SystemOut.log
  
[7/3/09 13:33:56:375 PDT] 00000019 ThreadMonitor W WSVR0605W: Thread "WebContainer : 0" (00000023) has been active for 66156 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.

  This message shows that Thread "WebContainer : 0" (00000023) is hanged so lets look at what is causing this thread to hang
  


• If you remember we configured hang detection policy so that it generates thread dump when a thread is hanged. So lets check profiles\AppSrv01\logs\server1\native_stderr.log file to find out if the thread dump was generated and if yes what is the location of the thread dump.
  
************* End Display Current Environment *************
JVMDUMP007I JVM Requesting Java Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090702.231210.3220.0001.txt'
JVMDUMP010I Java Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090702.231210.3220.0001.txt

  The native_stderr.log file has location of the javacore.


• Open the javacore.20090702.231210.3220.0001.txt file in Thread Dump Analyzer which is part of the IBM Support Assistant. And inside that take a look at stack trace of Thread "WebContainer : 0" (00000023) thread.
  
  
  As you can see the Thread "WebContainer : 0" (00000023) is executing the HangDetectionPolicy.doGet() method and it is executing Thred.Sleep(), so now we know what is causing the thread to hang

Hang Detection Policy

A common error in J2EE applications is a hung thread. A hung thread can result from a simple software defect (such as an infinite loop) or a more complex cause (for example, a resource deadlock). System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless an end user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.

Using the hang detection policy, you can specify a time that is too long for unit of work to complete, the thread monitor will monitor all the managed threads and check if any of the thread is running for more than threashold value if yes it will write a message in System.Out to let you know. The hang detection policy is on by default

Important Note: The hang detection policy only monitors managed threads such as web container threads or object request broker (ORB) threads(used for executing EJB). Unmanaged threads, which are created by the application are not monitored.

You can configure the hang detection policy and set following values.

• com.ibm.websphere.threadmonitor.threshold: The length of time (in seconds) in which a thread can be active before it is considered hung. Any thread that is detected as active for longer than this length of time is reported as hung. The default value is 10minutes or 600 seconds


• com.ibm.websphere.threadmonitor.interval: The frequency (in seconds) at which managed threads in the selected application server will be interrogated. Default value is 180 seconds or 3 minutes


• com.ibm.websphere.threadmonitor.false.alarm.threshold: The number of times (T) that false alarms can occur before automatically increasing the threshold. It is possible that a thread that is reported as hung eventually completes its work, resulting in a false alarm. A large number of these events indicates that the threshhold value is too small. The hang detection facility can automatically respond to this situation: For every T false alarms, the threshold T is increased by a factor of 1.5. Set the value to zero (or less) to disable the automatic adjustment. Default value is 100


• com.ibm.websphere.threadmonitor.dump.java: Set to true to cause a javacore to be created when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore can be analyzed to determine what the reported thread and other related threads are doing.