Debugging Tomcat memory leak using java VisualVM

Recently i was trying to debug a memory leak in the Web Application deployed in Apache Tomcat, i wanted to figure out how the JVM memory was doing. For that i decided to use Java VisualVM and it turns out that is pretty easy to use and very powerful.

1. I had the Apache Tomcat running on my machine, so i tested it if the application was running ok.
2. The next step was to execute jvisualvm command which is part of JDK.
3. It brought up Java VisualVM like this, as you can see it shows you every Java application that is running on your machine, including tomcat
   
4. Then double click on the Tomcat process and it opens the details like this, As you can see i dont have to set any JVM flags but could attach the VisualVM directly
   
5. The monitor tab gives you high level information about the VM like how its doing on memory and threads,.. etc. It also has Heap Dump button, when you click on it, That generates Heap Dump. By default the heap dump goes to /var/folders/.. folder. So right click on the Heap Dump related node and click on Save As and save it somewhere else on your disk. Then you can use Eclipse MAT to analyze the dump
   

Generating thread dump/java core using WSAdmin script

You can generate Thread Dump or Java core manually using the following WSAdmin Script

def generateThreadDump(serverName):
serverJVM = AdminControl.queryNames("type=JVM,process="+serverName+",*")
AdminControl.invoke(serverJVM,"dumpThreads")

generateThreadDump("server1")


The generateThreadDump() method takes server name and generates thread dump for that server. Once the thread dump is generated you can find out the location of the thread dump from native_stderr.log file. This is sample of the messages from my native_stderr.log


JVMDUMP007I JVM Requesting Java Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090705.212408.5480.0001.txt'
JVMDUMP010I Java Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090705.212408.5480.0001.txt


Manually generating heap dump

You can use the WSAdmin script to generate heapdump for the JVM. Use this Jython script to do that

def generateHeapDump(serverName):
serverJVM = AdminControl.queryNames("type=JVM,process="+serverName+",*")
print serverJVM
AdminControl.invoke(serverJVM,"generateHeapDump")


generateHeapDump("server1")


Just call the generateHeapDump() function with name of the server that you want to generate heap dump for.

Once this script is executed open the native_stderr.log file to find out the location of heap dump file. In my case i see these lines in the native_stderr.log file

JVMDUMP007I JVM Requesting Heap Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\heapdump.20090705.140924.8660.0001.phd'
JVMDUMP010I Heap Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\heapdump.20090705.140924.8660.0001.phd


You can open the file name ending with .phd in the HeapAnalyzer tool

IBM HeapAnalyzer

If you get OutOfMemory exception or your server is performing slow and the JVM Memory utilization is going up continously then you might want to take a look at JVM Memory Heap. THis is two step process first you should generate heap dump on your server and then analyze what all objects are there on the heap and if there is a memory leak

The IBM HeapAnalyzer tool lets you analyze the heap dump generated by WebSphere Application Server. This tool is shipped with IBM Support Assistant or you can download it from the Alpha works site. I tried using the HeapAnalyzer that is part of the ISA 4.1 but it kept crashing so i had to download it from Alpha Works Site

I wanted to learn how to use Heap Analyzer Tool to identify Memory Leaks, so i started by creating a MemoryLeakServlet and then i used the JMeter tool to generate load on this server. The load test ran for few minutes and then WAS started throwing OutOfMemory exception. Please note that when the WAS server throws OutOfMemory exception it generates Heap Dump.

Since i am using standalone version of Heap ANalyzer i had to start using the command line like this


The HeapANalyzer tool is very memory intensive so you should not run it on production and run it on a machine which has lot of memory.

Once the tool is opend select the .hpd file by clicking on Open and then selecting the heap dump file like this


The tool will take few minutes to analyze the heap data and then it will generate view like this.




As you can see the Reference Tree section is showing the objects on the heap in tree format. Now if you click on Subponea Leak Suspect button you will see that there is one suspected memory leak so select the button and it will take you the object which is suspected to have the memory leak.



As you can see it is showing the ArrayList object which is inside MemoryLeakServlet as memory leak.

IBM Pattern modeling and Analysis tool for Java

The IBM Pattern modeling and Analysis tool(PMAT) for Java lets analyze the data from garbage collection and in case of problems such as heap exhaustion (In case of OutOfMemory error) it also gives you recommendations. Please note one important point that the GC analysis tool alone cant help you locate the memory leak and fix it. You will have to use IBM Heap analyzer for that.

The PMAT tool is part of IBM Support Assistant 4.1, you can launch it by going to Launch Activities -< Analyze Problems. Then in the Tools Catalog select the IBM Pattern Modeling and Analysis tool and click on launch



It will ask you for the location of native_stderr.log file for your server. Select the file and click on Ok


Once opened it will show you the summary of gc analysis like this

• File name : Location and file name of verbosegc trace


• Number of verboseGC cycles : Number of JVM restart


• Number of Garbage Collections : GC frequency


• Number of Allocation failures : AF frequency


• First Garbage Collection : Timestamp of the first GC


• Last Garbage Collection : Timestamp of the last GC


• Number of Java heap exhaustion : Number of OutOfMemoryError


• Maximum AF overhead : Ratio of time spent in AF and time between AFs


• Number of 100% overhead : Number of AF overhead 100%


• Maximum size of Large Object Request : The largest object request and timestamp


• Number of Large Object Requests : Number of object request (>900KB)


• List of Java heap failure : Timestamp, Requested Java heap size,Type of failure and available Java heap size.

During the first few minutes of executing memory leak servlet. It did not throw OutOfMemory exception so in the recommendation it says that no memory exhaustion but there seems to be increase in java heap size, this can be taken as early warning sign of memory leak in the application.

After few minutes when the memory was exhausted and was started throwing OutOfMemory exception i looked at the gc analysis again in the PMAT tool and this is what i see


As you can see there are three errors related to java heap exhaustion that means the GC not able to free up memory and heap is full. One other thing is that whenever there is OutOfMemory error was would generate heap dump automatically. You can get location of the heapdump file from native_stderr.log file.


You can also look at the graphical memory analysis of the data by right clicking on the File and clicking Graphical view all


It will show you the graph of memory like this


As you can see the memory usage went really up some time after 12:33:35 and the red line shows the memory used and blue line show the heap freed after gc. As you can see red line is at near about 100 % and blue line at 0% that means memory is used and gc cant free up space.The vertical black dotted line where blue and red line terminate shows server restart.

Memory Leak Servlet

I wanted to learn how to identify the Memory Leaks in WAS and how to analyze the problem using
1) The IBM Pattern modeling and Analysis tool for Java
2) IBM HeapAnalyzer tool

So i started by creating a sample MemoryLeakServlet like this.

public class MemoryLeakServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
private static ArrayList st = new ArrayList();

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
ArrayList st1 = new ArrayList();
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
System.out.println("Inside MemoryLeakServlet.doGet() " + st.size());

st.add(st1);
response.setContentType("text/html");
response.getWriter().println("Hello from memory leak servlet");
}

}


As you can see the MemoryLeakServlet has one static ArrayList st and i keep adding data to it for every HTTP GET request.

Once that was done i used JMEter to create a simple performance test which kept making HTTP GET requests to MemoryLeakServlet until i got OutOfMemory leak error and during that time i kept looking at native_stderr.log file using PAM tool to see what type of data i see

Verbose garbage collection

When you turn the verbose garbage collection on, WAS will start printing garbage collection related information in your native_stderr.log file. You can turn the vebose garbage collection on using WAS Admin Console by going to Application servers < servername servers < Process Definition servers < Java Virtual Machine



On this screen check verbose garbage collection checkbox and restart the server for your changes to take effect. Once you enable the verbose garbage collection on the WAS server will start writing messages into native_stderr.log file every time it executes garbage collection.

These are couple of entries from my native_stderr.log file.


<af type="tenured" id="49" timestamp="Jul 05 13:13:17 2009" intervalms="32.872">
<minimum requested_bytes="16776" />
<time exclusiveaccessms="0.044" />
<tenured freebytes="3624584" totalbytes="116873216" percent="3" >
<soa freebytes="3624584" totalbytes="116873216" percent="3" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="49" totalid="49" intervalms="34.359">
<refs_cleared soft="0" threshold="32" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="107.399" sweep="1.583" compact="0.000" total="109.117" />
<tenured freebytes="43894704" totalbytes="116873216" percent="37" >
<soa freebytes="43894704" totalbytes="116873216" percent="37" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
</gc>
<tenured freebytes="43877928" totalbytes="116873216" percent="37" >
<soa freebytes="43877928" totalbytes="116873216" percent="37" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<time totalms="110.648" />
</af>

<af type="tenured" id="50" timestamp="Jul 05 13:13:18 2009" intervalms="232.708">
<minimum requested_bytes="32" />
<time exclusiveaccessms="0.031" />
<tenured freebytes="0" totalbytes="116873216" percent="0" >
<soa freebytes="0" totalbytes="116873216" percent="0" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="50" totalid="50" intervalms="234.140">
<classloadersunloaded count="13" timetakenms="48.694" />
<expansion type="tenured" amount="19868672" newsize="136741888" timetaken="0.152" reason="excessive time being spent in gc" gctimepercent="49" />
<refs_cleared soft="0" threshold="32" weak="3" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="118.982" sweep="2.639" compact="0.000" total="170.899" />
<tenured freebytes="55392816" totalbytes="136741888" percent="40" >
<soa freebytes="55392816" totalbytes="136741888" percent="40" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
</gc>
<tenured freebytes="55392176" totalbytes="136741888" percent="40" >
<soa freebytes="55392176" totalbytes="136741888" percent="40" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<time totalms="172.525" />
</af>


The way IBM JDK works is if it is not able to allocate a memory then it will execute garbage collection to free up the memory. The J9 VM used in WAS 6.1 generates one <af> element every time a garbage collection works.

The af element has following elements

• type:


• id: The id represents how many times the gc was executed


• intervalms: The time in ms since last time gc was executed


• timestamp: time of gc

The minimum represents the number of bytes that were requested and JVM couldnot allocate them so it had to trigger garbage collection cycle.

The af element has 3 main child elements first tenured element has data about the tenured memory position before gc then gc element represents data about what happened during gc, such as time spent in mark, sweep and compact phases, The second tenured element represents the position of tenured memory after gc.


The IBM Support assistance has IBM Pattern modeling and Analysis tool for Java Garbage collection tool that can be used to analyze the garbage collection.

Hang Detection Policy Example

I wanted to try hang detection policy so i decided to create a sample application that will first create hang and then i will use the WAS tools to debug the issue. I followed these steps

• Create a sample HangDetectionServlet as shown in the listing
  
public class HangDetectionServlet extends HttpServlet {
private static final long serialVersionUID = 1L;

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
DateFormat d = new SimpleDateFormat("mm:ss:SS");
response.getWriter().println("Entering HangDetectionServlet.doGet() " + d.format(new Date()));
try {
Thread.sleep(250000);
} catch (InterruptedException e) {
e.printStackTrace();
}
response.getWriter().println("Exiting HangDetectionServlet.doGet() " + d.format(new Date()));
}

}

  This servlet is very simple, when it gets HTTP GET request it puts the current thread in sleep for 250 seconds.
  


• By default WAS marks a thread has hanged thread if it is running for more than 10 minutes. So our HangDetectionServlet will not be marked as hanged. So in the next step we will change the hang detection policy on WAS so that if the thread is running for more than 60 seconds/ 1 minute it will be considered hanged.


• You can set Hang Detection policy on WAS using WAS Admin Console. First Login into WAS admin console and then go to Servers < Application Servers < server_name. Then under Server Infrastructure go to Administration -> Custom Properties.
  
  


• Set hang detection policies like this
  
  By setting value of com.ibm.websphere.threadmonitor.threshold to 60 i am saying that if the thread is running for more than 60 seconds then it should be considered hanged and setting com.ibm.websphere.threadmonitor.dump.java to true means when application server detects hanged thread it should generate thread dump in addition to writing message in the SysetemOut.log. Setting value of com.ibm.websphere.threadmonitor.interval to 60 means saying that the thread monitor should run every 60 seconds to check for hanged thread. After setting these values restart the server for changes to take effect


• Now deploy the HangDetectionServlet on your server and access it, after couple of minutes i could see this message in the SystemOut.log
  
[7/3/09 13:33:56:375 PDT] 00000019 ThreadMonitor W WSVR0605W: Thread "WebContainer : 0" (00000023) has been active for 66156 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.

  This message shows that Thread "WebContainer : 0" (00000023) is hanged so lets look at what is causing this thread to hang
  


• If you remember we configured hang detection policy so that it generates thread dump when a thread is hanged. So lets check profiles\AppSrv01\logs\server1\native_stderr.log file to find out if the thread dump was generated and if yes what is the location of the thread dump.
  
************* End Display Current Environment *************
JVMDUMP007I JVM Requesting Java Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090702.231210.3220.0001.txt'
JVMDUMP010I Java Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090702.231210.3220.0001.txt

  The native_stderr.log file has location of the javacore.


• Open the javacore.20090702.231210.3220.0001.txt file in Thread Dump Analyzer which is part of the IBM Support Assistant. And inside that take a look at stack trace of Thread "WebContainer : 0" (00000023) thread.
  
  
  As you can see the Thread "WebContainer : 0" (00000023) is executing the HangDetectionPolicy.doGet() method and it is executing Thred.Sleep(), so now we know what is causing the thread to hang

Hang Detection Policy

A common error in J2EE applications is a hung thread. A hung thread can result from a simple software defect (such as an infinite loop) or a more complex cause (for example, a resource deadlock). System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless an end user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.

Using the hang detection policy, you can specify a time that is too long for unit of work to complete, the thread monitor will monitor all the managed threads and check if any of the thread is running for more than threashold value if yes it will write a message in System.Out to let you know. The hang detection policy is on by default

Important Note: The hang detection policy only monitors managed threads such as web container threads or object request broker (ORB) threads(used for executing EJB). Unmanaged threads, which are created by the application are not monitored.

You can configure the hang detection policy and set following values.

• com.ibm.websphere.threadmonitor.threshold: The length of time (in seconds) in which a thread can be active before it is considered hung. Any thread that is detected as active for longer than this length of time is reported as hung. The default value is 10minutes or 600 seconds


• com.ibm.websphere.threadmonitor.interval: The frequency (in seconds) at which managed threads in the selected application server will be interrogated. Default value is 180 seconds or 3 minutes


• com.ibm.websphere.threadmonitor.false.alarm.threshold: The number of times (T) that false alarms can occur before automatically increasing the threshold. It is possible that a thread that is reported as hung eventually completes its work, resulting in a false alarm. A large number of these events indicates that the threshhold value is too small. The hang detection facility can automatically respond to this situation: For every T false alarms, the threshold T is increased by a factor of 1.5. Set the value to zero (or less) to disable the automatic adjustment. Default value is 100


• com.ibm.websphere.threadmonitor.dump.java: Set to true to cause a javacore to be created when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore can be analyzed to determine what the reported thread and other related threads are doing.

Using Thread dump analyzer to detect server hang/ Server Crash

Recently i was debugging one server crash problem during performance testing. So what was happening is the code was running ok during manual testing but once we started performance testing and have 200 users hitting portal every minute after 30 minutes of so server was getting hanged and stopped responding. When i looked at the SystemOut i could see the WebContainer thread hanged message (I think WebSphere VE is generating that alert).

So when the server was hanged we took javacore dump on portal node using kill -3 <serverprocessid>. Then i took the javacore.*.txt file and analyzed it using the Thread Dump analyzer following these steps.

• Start the thread dump analyzer and open the javacore.*.txt file. Thread dump analyzer will take couple of minutes then display this UI.
  
  
  


• As you can see the Monitor column is showing 3 open monitors. Monitor is used when your saying using synchronized block or function any where. If you have say static synchronized method then only one thread can enter in it and others would wait to get monitor for that thread


• Right click on the javacore file and click on Thread details. The next screen will show details of all the threads running. YOu can see that there are no threads in the deadlock but 255 threads are blocked. That means 255 threads are waiting for something.
  
  


• Now you can go back to the main screen and right click on the javacore and say Monitor detail. It will open a tree view like this
  
  


• This view displays which thread is currently running and has monitor and who all is waiting for that monitor. In my case Non-deferrable alarm: 3 thread is running and it owns lock on com/ibm/ws/cache/Cache@ object, which is value of Monitor field on the right hand size. Then under that thread is list of threads waiting to get lock on the cache object. The WebContainer 67 thread is waiting for the lock on that object and there are 218 threads waiting for Monitor that WebContainer 67 owns.


• Conclusion of this issue was that Non-deferrable thread is doing something to block all the web container threads. I looked at the stack trace for that thread on the right hand side to figure out it is doing some database operation. When i scrolled little down i could see my hibernate function that was making JDBC query. This JDBC query was taking really long to return results so i made changes to fix it and that solved my server hang issue

Application server or portal server hangs

It might happen that your application or portal server is hanging or getting terminated unexpectedly or other case is the server is taking really long time to respond. In these cases you might want to generate JavaDump to see what different threads are doing and then analyze the data to find the problem. Although a Javacore or javadump can occur in Solaris JVMs, much of the content of the Javacore is added by IBM and, so, is present only in IBM JVMs.

Important: If there is a steady increase in servlet engine threads in use, review application synchronized code blocks for possible deadlock conditions.

Generating JavaDump manually

By triggering Java thread dump or javacore when a process does not respond,its possible to collect diagnostic information related to the JVM captured at particular point of execution.The code that creates a JavaCore or javadump is part of the JVM. By defautl a javacore occurs when the JVM terminates unexpectedly or it can be triggered by sending specific signal to the JVM.

We can generate thread dumps by first finding out the process id of your server from the .pid file in the logs folder for that server then you have two options
You can either execute kill -3 <processid>, this command would generate thread dump but wont terminate the JVM or you can execute the kill -11 <processid> command this will generate the thread dump and kill the java process. On the Windows platform, you can use keyboard combinations like Ctrl-Break to send signal 3 to a process running in a command line window. THe resultant javacore file would be close 1MB or little more than that in size depending on no. of threads that you have.

Other option for generating thread dump is by executing this wsadmin script. In this case i am generating dump of WebSphere_Portal server, thats why i am passing process=WebSphere_Portal. Use Server Name of the server that you want to generate thread dump for.

jvm=AdminControl.completeObjectName('type=JVM,process=WebSphere_Portal,*')
AdminControl.invoke(jvm,'dumpThreads')

Once you execute this command you will see javacore.*.txt file in your profile directory. In case of WebSphere_Portal it will be in the wp_profile directory.

Once you have the JavaCore file you can either open it in text editor and try to understand it or you can use one of the tools that are available from the IBM support site to read and analyze this file for you.

What Is a Thread Dump?

A Java thread dump is the one of the traces/dumps that JVM provides to help diagnosis a hang, deadlock, or monitor contention issue. It contains diagnostic information related to the JVM and a Java application captured at a point during execution. For example, the information can be a list of all the threads that run on a Java virtual machine. Usually thread dumps have more than just thread information. Thread dumps can produce information about the operating system, application environment, threads, stacks, locks, and memory. IBM Thread Dumps/Javacores provide much more information like core interface, data conversion/unicode, class, diagnosis, execution management/thread management, lock, execution engine, mixed mode interpreter, JIT (Just-in-Time compiler), storage/JVM heap, and hardware portability interface/extended hardware portability interface.
The contents and formats of Java thread dumps depend on the platform you're running on and the JVM provider.
Since a thread dump's format isn't a part of the SDK specification, each SDK vendor provides a unique thread dump format and its own JVM information. Currently IBM Thread and Monitor Dump Analyzer for Java understands IBM, Solaris, and HP-UX Java thread dump formats.

Thread dump analyzer

You can download the IBM Thread and Monitor dump analyzer tool from IBM Alphaworks site and then open it by executing following command

java -Xmx[heap size] -jar jca.jar_

The thread dump analyzer is also available as part of IBM Support Assistant. You can launch it from IBM support assistant by going to Launch Activity -> Analyze Problem then switching to Tools tab. On this tab select IBM Thread and Dump Analyzer for Java and click on launch to launch the tool



Once the tool is started on the first screen select File -> Open Thread Dumps and select the java core file that you just generated like this



The Thread Dump analyzer will take few minutes to analyze the javacore and once it is done analyzing it will display name of the javacore when you select that name it will display details about the process in the pan below the name. You can right click and select Thread Details



It will take you to the next screen that will display what all threads are running and what is state of the threads. Check if you can see any deadlocks here or threads that are in blocked state waiting for some operation to complete



In my case there are no deadlocks but large no. of threads are blocked. So i will have to find out why threads are blocked. You can find information on how to use Thread dump analyzer in the help section.