Increasing no. of parallel downloads from same host

Most of the web 2.0 sites are rich in nature, by that i mean they have more than 30 resources( CSS, images, .js files). One disadvantage of the rich site is that it slows down the end user experience.

If you look at the portal site in firebug Net panel you will notice that the downloading of basic HTML takes less than 20-30 % time but major chunk of time is spent in downloading resources.

Basic problem is that the browsers use limited no. of connections for downloading resources from one host.Firefox 3.6, IE 8.0 use 6 and older browsers such as IE 6 or IE 7 use only 2 connections. So if your site has say 30 resources IE6 wont download all of them at the same time instead it will download 2 resources at a time. In one of the tests that i done i realized that the same site takes 1.5-2 times as long to render on IE 6 compared to IE 8 or Firefox

Both IE and Firefox allow you to change no. of parallel connections that should be used to download. This link has information on How do I configure Internet Explorer to download more than two files at one time?

You can make similar changes in Firefox by going to about:config and then changing value of network.http.max-persistent-http-connection-per-server.




We cant expect users to change the registry settings or change firefox configuration for your site. But these are nice to know tricks when debugging performance issues

64 bit performance tuning tips

Applications that run on the WebSphere Application Server 64-bit version can see significant performance gains if they can leverage the support for large heaps. To exploit the large heap, you can use dynamic cache in WebSphere Application Server. The process to tune dynamic cache remains the same for the 64-bit version as for the 32-bit version. However, all memory references are also 64-bit wide on the 64-bit version, which is twice the size of memory references on the 32-bit version. This size difference causes an increase in the memory requirements of the applications when deployed on 64-bit, as compared to 32-bit.

Web Container Transport channel settings

The Web Container transport channel lets you configure the write size of the buffer. In the Tunning Transport Channel service, we covered the basics of what is channel service. In that page click on the TCP Inbound Channel link and you will get this page




If multiple writes are required to handle responses to the client, change the value specified for the Write buffer size parameter to a value that is more appropriate for your clients. The Write buffer size parameter controls the maximum amount of data per thread that the Web container buffers before sending the request on for processing. The default value is 32768 bytes, which is sufficient for most applications. If the size of a response is greater than the size of the write buffer, the response is chunked and written back in multiple TCP writes. If you need to change the value specified for this parameter, make sure the new value enables most requests to be written out in a single write. To determined an appropriate value for this parameter, look at the size of the pages that are returned and add some additional bytes to account for the HTTP headers.

Enabling Keep Alive in WAS

The HTTP Channel service lets you configure behavior of the HTTP Channel, things like if you want to keep the HTTP connective alive In the Tunning Transport Channel service, we covered the basics of what is channel service. In that page click on the HTTP Inbound Channel link and you will get this page




The Use Persistent (keep-alive) connections settings controls whether or not connections are left open between requests. Leaving the connections open can save setup and teardown costs of sockets if your workload has clients that send multiple requests. THe default value is true and which is optimal setting in most cases.If your clients only send single requests over substantially long periods of time, it is probably better to disable this option and close the connections right away rather than to have the HTTP transport channel setup the timeouts to close the connection at some later time.

TCP Channel tunning

You might want to tune TCP Channel related settings if say your backend is slow and the CPU is unused. In the Tunning Transport Channel service, we covered the basics of what is channel service. In that page click on the TCP Inbound Channel link and you will get this page




The most important setting on this page is the thread pool assigned to the channel. Each TCP transport channel is assigned to a particular thread pool. Thread pools can be shared between one or more TCP transport channels as well as with other components. The default settings for a TCP transport channel is to have all HTTP based traffic assigned to the WebContainer thread pool and all other traffic assigned to the Default thread pool.

Tune the size of your thread pools. By default, a thread pool can have a minimum of 10 threads and a maximum of 50 maximum threads. To adjust these values, click on Thread pools < threadpool_name and adjust the values specified for the Minimum Size and Maximum Size parameters for that thread pool. Typical applications usually do not need more than 10 threads per processor. One exception is if there is some off server condition, such as a very slow backend request, that causes a server thread to wait for the backend request to complete. In such a case, CPU usage is usually low and increasing the workload does not increase CPU throughput. Thread dumps show nearly all threads in a call out to the backend resource. If this condition exists, and the backend is tuned correctly, try increasing the minimum number of threads in the pooll until you see improvements in throughput and thread dumps show threads in other areas of the runtime besides the backend call. The setting for the Grow as needed parameter should not be changed unless your backend is prone to hanging for long periods of time. This condition might indicate that all of your runtime threads are blocked waiting for the backend instead of processing other work that does not involve the hung backend


Leave the Maximum open connections parameter set to the default value. This parameter controls the maximum number of connections that are available for a server’s use. It should be left at the default value of 20000, which is the maximum number of connections allowed

Tuning transport channel services

The transport channel service manage client connections and I/O processing for HTTP and JMS requests. These services provide a highly scalable foundation to WebSphere Application Server request processing. Java NIO based architecture has limitations in terms of performance, scalability and end user usability. Therefore, integration of true asynchronous I/O is implemented. This implementation provides significant benefits in usability, reduces the complexity of I/O processing and reduces that amount of performance tuning you have to perform.Changing the default values for settings on one or more of the transport channels associated with a transport chain can improve the performance of that chain.

I copied this diagram from IBM Redbook, it explains how the Transport channel service works.




You can access the the transportation related configuration in WAS Admin Console by going to Servers - server_name -> Web Container settings - Web Container transport chains, then select WCInboundDefault, which is the virtual host for handling all the requests to web application on that server.



As you can see there is a hyper link for configuring settings related to each of the three channels

Tuning Java virtual Machine

WebSphere application server is a Java based server and requires a Java virtual machine to run the server and applications deployed on it. Tunning underlying Java virtual machine can play a major role in tuning performance. The WebSphere Application Server runs on different JVMS. In most of the cases it is IBM JDK, but on Solaris it is Sun JDK,.. What parameters you can use and the syntax to set there values depends on the type of JVM your tuning.

The parameters that can be used for tuning JVM can be divided into following three types and you can tune all three of them from WAS Admin Console this page

Java memory or heap tuning: Java memory or heap controls the amount of memory that is allocated for use by individual application server instance. You can change the JVM both initial and maximum heap size. Increasing this parameter increases the memory available to the application server, and reduces the frequency of garbage collection. Increasing this setting can improve server response time and throughput. However, increasing this setting also increases the duration of a garbage collection when it does occur. This setting should never be increased above the system memory available for the application server instance. Increasing the setting above the available system memory can cause system paging and a significant decrease in performance. Tuning value of initial heap size reduces the overhead of garbage collection, which improves server response time and throughput. You can change the initial or maximum heap size by changing value of "Initial heap size" and "Maximum heap size" fields.

Garbage collection tuning: When the JVM cannot allocate an object from the current heap because of lack of contiguous space, the garbage collector is invoked to reclaim memory from Java objects that are no longer being used. Each JVM vendor provides unique garbage collector policies and tuning parameters. You can set the garbage collection policy by passing it as command line parameter using -X option suitable for your JVM. The generational garbage collection is the best performing garbage collection policy, Also by default, the JVM unloads a class from memory whenever there are no live instances of that class left. Class unloading can degrade performance. Turning off class garbage collection eliminates the overhead of loading and unloading the same class multiple times.

Start up versus runtime performance optimization: In some environments, such as development environment, it is more important to optimize the startup performance of your application server rathar than the runtime performance. In production or test environment it is more important to optimize the runtime performance than startup performance.The Java JIT compiler has a big impact on whether startup or runtime performance is optimized. The initial optimization level that the compiler uses influences the length of time it takes to compile a class method, and the length of time it takes to start the server. For faster startups, you should reduce the initial optimization level that the compiler uses. However if you reduce the initial optimization level, the runtime performance of your applications might be degraded because the class methods are now compiled at a lower optimization level. The share classes option of the IBM Java 2 Runtime Environment (J2RE) Version 1.5.0 lets you share classes in a cache. Sharing classes in a cache can improve startup time and reduce memory footprint. Processes, such as application servers, node agents, and deployment managers, can use the share classes option.

Generating thread dump/java core using WSAdmin script

You can generate Thread Dump or Java core manually using the following WSAdmin Script

def generateThreadDump(serverName):
serverJVM = AdminControl.queryNames("type=JVM,process="+serverName+",*")
AdminControl.invoke(serverJVM,"dumpThreads")

generateThreadDump("server1")


The generateThreadDump() method takes server name and generates thread dump for that server. Once the thread dump is generated you can find out the location of the thread dump from native_stderr.log file. This is sample of the messages from my native_stderr.log


JVMDUMP007I JVM Requesting Java Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090705.212408.5480.0001.txt'
JVMDUMP010I Java Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090705.212408.5480.0001.txt


Manually generating heap dump

You can use the WSAdmin script to generate heapdump for the JVM. Use this Jython script to do that

def generateHeapDump(serverName):
serverJVM = AdminControl.queryNames("type=JVM,process="+serverName+",*")
print serverJVM
AdminControl.invoke(serverJVM,"generateHeapDump")


generateHeapDump("server1")


Just call the generateHeapDump() function with name of the server that you want to generate heap dump for.

Once this script is executed open the native_stderr.log file to find out the location of heap dump file. In my case i see these lines in the native_stderr.log file

JVMDUMP007I JVM Requesting Heap Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\heapdump.20090705.140924.8660.0001.phd'
JVMDUMP010I Heap Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\heapdump.20090705.140924.8660.0001.phd


You can open the file name ending with .phd in the HeapAnalyzer tool

IBM HeapAnalyzer

If you get OutOfMemory exception or your server is performing slow and the JVM Memory utilization is going up continously then you might want to take a look at JVM Memory Heap. THis is two step process first you should generate heap dump on your server and then analyze what all objects are there on the heap and if there is a memory leak

The IBM HeapAnalyzer tool lets you analyze the heap dump generated by WebSphere Application Server. This tool is shipped with IBM Support Assistant or you can download it from the Alpha works site. I tried using the HeapAnalyzer that is part of the ISA 4.1 but it kept crashing so i had to download it from Alpha Works Site

I wanted to learn how to use Heap Analyzer Tool to identify Memory Leaks, so i started by creating a MemoryLeakServlet and then i used the JMeter tool to generate load on this server. The load test ran for few minutes and then WAS started throwing OutOfMemory exception. Please note that when the WAS server throws OutOfMemory exception it generates Heap Dump.

Since i am using standalone version of Heap ANalyzer i had to start using the command line like this


The HeapANalyzer tool is very memory intensive so you should not run it on production and run it on a machine which has lot of memory.

Once the tool is opend select the .hpd file by clicking on Open and then selecting the heap dump file like this


The tool will take few minutes to analyze the heap data and then it will generate view like this.




As you can see the Reference Tree section is showing the objects on the heap in tree format. Now if you click on Subponea Leak Suspect button you will see that there is one suspected memory leak so select the button and it will take you the object which is suspected to have the memory leak.



As you can see it is showing the ArrayList object which is inside MemoryLeakServlet as memory leak.

IBM Pattern modeling and Analysis tool for Java

The IBM Pattern modeling and Analysis tool(PMAT) for Java lets analyze the data from garbage collection and in case of problems such as heap exhaustion (In case of OutOfMemory error) it also gives you recommendations. Please note one important point that the GC analysis tool alone cant help you locate the memory leak and fix it. You will have to use IBM Heap analyzer for that.

The PMAT tool is part of IBM Support Assistant 4.1, you can launch it by going to Launch Activities -< Analyze Problems. Then in the Tools Catalog select the IBM Pattern Modeling and Analysis tool and click on launch



It will ask you for the location of native_stderr.log file for your server. Select the file and click on Ok


Once opened it will show you the summary of gc analysis like this

• File name : Location and file name of verbosegc trace


• Number of verboseGC cycles : Number of JVM restart


• Number of Garbage Collections : GC frequency


• Number of Allocation failures : AF frequency


• First Garbage Collection : Timestamp of the first GC


• Last Garbage Collection : Timestamp of the last GC


• Number of Java heap exhaustion : Number of OutOfMemoryError


• Maximum AF overhead : Ratio of time spent in AF and time between AFs


• Number of 100% overhead : Number of AF overhead 100%


• Maximum size of Large Object Request : The largest object request and timestamp


• Number of Large Object Requests : Number of object request (>900KB)


• List of Java heap failure : Timestamp, Requested Java heap size,Type of failure and available Java heap size.

During the first few minutes of executing memory leak servlet. It did not throw OutOfMemory exception so in the recommendation it says that no memory exhaustion but there seems to be increase in java heap size, this can be taken as early warning sign of memory leak in the application.

After few minutes when the memory was exhausted and was started throwing OutOfMemory exception i looked at the gc analysis again in the PMAT tool and this is what i see


As you can see there are three errors related to java heap exhaustion that means the GC not able to free up memory and heap is full. One other thing is that whenever there is OutOfMemory error was would generate heap dump automatically. You can get location of the heapdump file from native_stderr.log file.


You can also look at the graphical memory analysis of the data by right clicking on the File and clicking Graphical view all


It will show you the graph of memory like this


As you can see the memory usage went really up some time after 12:33:35 and the red line shows the memory used and blue line show the heap freed after gc. As you can see red line is at near about 100 % and blue line at 0% that means memory is used and gc cant free up space.The vertical black dotted line where blue and red line terminate shows server restart.

Memory Leak Servlet

I wanted to learn how to identify the Memory Leaks in WAS and how to analyze the problem using
1) The IBM Pattern modeling and Analysis tool for Java
2) IBM HeapAnalyzer tool

So i started by creating a sample MemoryLeakServlet like this.

public class MemoryLeakServlet extends HttpServlet {
private static final long serialVersionUID = 1L;
private static ArrayList st = new ArrayList();

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
ArrayList st1 = new ArrayList();
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
st1.add("This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string This is a big string ");
System.out.println("Inside MemoryLeakServlet.doGet() " + st.size());

st.add(st1);
response.setContentType("text/html");
response.getWriter().println("Hello from memory leak servlet");
}

}


As you can see the MemoryLeakServlet has one static ArrayList st and i keep adding data to it for every HTTP GET request.

Once that was done i used JMEter to create a simple performance test which kept making HTTP GET requests to MemoryLeakServlet until i got OutOfMemory leak error and during that time i kept looking at native_stderr.log file using PAM tool to see what type of data i see

Verbose garbage collection

When you turn the verbose garbage collection on, WAS will start printing garbage collection related information in your native_stderr.log file. You can turn the vebose garbage collection on using WAS Admin Console by going to Application servers < servername servers < Process Definition servers < Java Virtual Machine



On this screen check verbose garbage collection checkbox and restart the server for your changes to take effect. Once you enable the verbose garbage collection on the WAS server will start writing messages into native_stderr.log file every time it executes garbage collection.

These are couple of entries from my native_stderr.log file.


<af type="tenured" id="49" timestamp="Jul 05 13:13:17 2009" intervalms="32.872">
<minimum requested_bytes="16776" />
<time exclusiveaccessms="0.044" />
<tenured freebytes="3624584" totalbytes="116873216" percent="3" >
<soa freebytes="3624584" totalbytes="116873216" percent="3" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="49" totalid="49" intervalms="34.359">
<refs_cleared soft="0" threshold="32" weak="0" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="107.399" sweep="1.583" compact="0.000" total="109.117" />
<tenured freebytes="43894704" totalbytes="116873216" percent="37" >
<soa freebytes="43894704" totalbytes="116873216" percent="37" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
</gc>
<tenured freebytes="43877928" totalbytes="116873216" percent="37" >
<soa freebytes="43877928" totalbytes="116873216" percent="37" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<time totalms="110.648" />
</af>

<af type="tenured" id="50" timestamp="Jul 05 13:13:18 2009" intervalms="232.708">
<minimum requested_bytes="32" />
<time exclusiveaccessms="0.031" />
<tenured freebytes="0" totalbytes="116873216" percent="0" >
<soa freebytes="0" totalbytes="116873216" percent="0" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<gc type="global" id="50" totalid="50" intervalms="234.140">
<classloadersunloaded count="13" timetakenms="48.694" />
<expansion type="tenured" amount="19868672" newsize="136741888" timetaken="0.152" reason="excessive time being spent in gc" gctimepercent="49" />
<refs_cleared soft="0" threshold="32" weak="3" phantom="0" />
<finalization objectsqueued="0" />
<timesms mark="118.982" sweep="2.639" compact="0.000" total="170.899" />
<tenured freebytes="55392816" totalbytes="136741888" percent="40" >
<soa freebytes="55392816" totalbytes="136741888" percent="40" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
</gc>
<tenured freebytes="55392176" totalbytes="136741888" percent="40" >
<soa freebytes="55392176" totalbytes="136741888" percent="40" />
<loa freebytes="0" totalbytes="0" percent="0" />
</tenured>
<time totalms="172.525" />
</af>


The way IBM JDK works is if it is not able to allocate a memory then it will execute garbage collection to free up the memory. The J9 VM used in WAS 6.1 generates one <af> element every time a garbage collection works.

The af element has following elements

• type:


• id: The id represents how many times the gc was executed


• intervalms: The time in ms since last time gc was executed


• timestamp: time of gc

The minimum represents the number of bytes that were requested and JVM couldnot allocate them so it had to trigger garbage collection cycle.

The af element has 3 main child elements first tenured element has data about the tenured memory position before gc then gc element represents data about what happened during gc, such as time spent in mark, sweep and compact phases, The second tenured element represents the position of tenured memory after gc.


The IBM Support assistance has IBM Pattern modeling and Analysis tool for Java Garbage collection tool that can be used to analyze the garbage collection.

Hang Detection Policy Example

I wanted to try hang detection policy so i decided to create a sample application that will first create hang and then i will use the WAS tools to debug the issue. I followed these steps

• Create a sample HangDetectionServlet as shown in the listing
  
public class HangDetectionServlet extends HttpServlet {
private static final long serialVersionUID = 1L;

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
DateFormat d = new SimpleDateFormat("mm:ss:SS");
response.getWriter().println("Entering HangDetectionServlet.doGet() " + d.format(new Date()));
try {
Thread.sleep(250000);
} catch (InterruptedException e) {
e.printStackTrace();
}
response.getWriter().println("Exiting HangDetectionServlet.doGet() " + d.format(new Date()));
}

}

  This servlet is very simple, when it gets HTTP GET request it puts the current thread in sleep for 250 seconds.
  


• By default WAS marks a thread has hanged thread if it is running for more than 10 minutes. So our HangDetectionServlet will not be marked as hanged. So in the next step we will change the hang detection policy on WAS so that if the thread is running for more than 60 seconds/ 1 minute it will be considered hanged.


• You can set Hang Detection policy on WAS using WAS Admin Console. First Login into WAS admin console and then go to Servers < Application Servers < server_name. Then under Server Infrastructure go to Administration -> Custom Properties.
  
  


• Set hang detection policies like this
  
  By setting value of com.ibm.websphere.threadmonitor.threshold to 60 i am saying that if the thread is running for more than 60 seconds then it should be considered hanged and setting com.ibm.websphere.threadmonitor.dump.java to true means when application server detects hanged thread it should generate thread dump in addition to writing message in the SysetemOut.log. Setting value of com.ibm.websphere.threadmonitor.interval to 60 means saying that the thread monitor should run every 60 seconds to check for hanged thread. After setting these values restart the server for changes to take effect


• Now deploy the HangDetectionServlet on your server and access it, after couple of minutes i could see this message in the SystemOut.log
  
[7/3/09 13:33:56:375 PDT] 00000019 ThreadMonitor W WSVR0605W: Thread "WebContainer : 0" (00000023) has been active for 66156 milliseconds and may be hung. There is/are 1 thread(s) in total in the server that may be hung.

  This message shows that Thread "WebContainer : 0" (00000023) is hanged so lets look at what is causing this thread to hang
  


• If you remember we configured hang detection policy so that it generates thread dump when a thread is hanged. So lets check profiles\AppSrv01\logs\server1\native_stderr.log file to find out if the thread dump was generated and if yes what is the location of the thread dump.
  
************* End Display Current Environment *************
JVMDUMP007I JVM Requesting Java Dump using 'C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090702.231210.3220.0001.txt'
JVMDUMP010I Java Dump written to C:\Cert\WebSphere\AppServer\profiles\AppSrv01\javacore.20090702.231210.3220.0001.txt

  The native_stderr.log file has location of the javacore.


• Open the javacore.20090702.231210.3220.0001.txt file in Thread Dump Analyzer which is part of the IBM Support Assistant. And inside that take a look at stack trace of Thread "WebContainer : 0" (00000023) thread.
  
  
  As you can see the Thread "WebContainer : 0" (00000023) is executing the HangDetectionPolicy.doGet() method and it is executing Thred.Sleep(), so now we know what is causing the thread to hang

Hang Detection Policy

A common error in J2EE applications is a hung thread. A hung thread can result from a simple software defect (such as an infinite loop) or a more complex cause (for example, a resource deadlock). System resources, such as CPU time, might be consumed by this hung transaction when threads run unbounded code paths, such as when the code is running in an infinite loop. Alternately, a system can become unresponsive even though all resources are idle, as in a deadlock scenario. Unless an end user or a monitoring tool reports the problem, the system may remain in this degraded state indefinitely.

Using the hang detection policy, you can specify a time that is too long for unit of work to complete, the thread monitor will monitor all the managed threads and check if any of the thread is running for more than threashold value if yes it will write a message in System.Out to let you know. The hang detection policy is on by default

Important Note: The hang detection policy only monitors managed threads such as web container threads or object request broker (ORB) threads(used for executing EJB). Unmanaged threads, which are created by the application are not monitored.

You can configure the hang detection policy and set following values.

• com.ibm.websphere.threadmonitor.threshold: The length of time (in seconds) in which a thread can be active before it is considered hung. Any thread that is detected as active for longer than this length of time is reported as hung. The default value is 10minutes or 600 seconds


• com.ibm.websphere.threadmonitor.interval: The frequency (in seconds) at which managed threads in the selected application server will be interrogated. Default value is 180 seconds or 3 minutes


• com.ibm.websphere.threadmonitor.false.alarm.threshold: The number of times (T) that false alarms can occur before automatically increasing the threshold. It is possible that a thread that is reported as hung eventually completes its work, resulting in a false alarm. A large number of these events indicates that the threshhold value is too small. The hang detection facility can automatically respond to this situation: For every T false alarms, the threshold T is increased by a factor of 1.5. Set the value to zero (or less) to disable the automatic adjustment. Default value is 100


• com.ibm.websphere.threadmonitor.dump.java: Set to true to cause a javacore to be created when a hung thread is detected and a WSVR0605W message is printed. The threads section of the javacore can be analyzed to determine what the reported thread and other related threads are doing.

Debugging Connection leaks

One of the common problems that we uncover during performance testing is JDBC Connection leak. When a J2EE project is using normal JDBC code instead of OR-Mapping framework such Hibernate or Spring there is a good chance that they run into issues that connection was opened but not closed so there is a connection leak.

The code works fine in development environment but in performance environment after some time you start seeing this error Timed Out waiting for connection. This shows problem that some part of your business logic is trying to open a database connection but it is not able to get it.

[7/2/09 9:53:21:411 PDT] 00000085 AdapterUtil > toSQLException Entry
com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: Connection not available, Timed out waiting for 180004
at com.ibm.ejs.j2c.FreePool.createOrWaitForConnection(FreePool.java:1470)
at com.ibm.ejs.j2c.PoolManager.reserve(PoolManager.java:2141)
at com.ibm.ejs.j2c.ConnectionManager.allocateMCWrapper(ConnectionManager.java:843)
at com.ibm.ejs.j2c.ConnectionManager.allocateConnection(ConnectionManager.java:582)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:431)
at com.ibm.ws.rsadapter.jdbc.WSJdbcDataSource.getConnection(WSJdbcDataSource.java:400)
at com.wpcertification.jdbc.ConnectionLeakServlet.doGet(ConnectionLeakServlet.java:41)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:966)
at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:478)
at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:463)
at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3107)
at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:238)
at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:811)
at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1425)
at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:92)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:465)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:394)
at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:274)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:152)
at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:213)
at com.ibm.io.async.AbstractAsyncFuture.fireCompletionActions(AbstractAsyncFuture.java:195)
at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136)
at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:193)
at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:725)
at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:847)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1498)


The IBM Support Assistant tool provides "Database connection pool analyzer" that you can use to debug the connection pool problems. Follow these steps to work with the tool.

• First enable the JDBC Connection pool related trace by enabling trace for the following trace string
  
WAS.database=finest:WAS.j2c=finest:RRA=finest

  


• Run your load test case and let it generate the com.ibm.websphere.ce.j2c.ConnectionWaitTimeoutException: error.


• Start the IBM Support assistant and then go to Launch Activity -< Analyzer Problem -< Tools.


• Select the Database Connection Pool Analyzer for IBM WebSPhere Application Server and click on launch button. It will ask you for name of the trace file, so select the trace file from your app server that has Connection pool related trace.
  
  


• It will open the tool with information on all the data sources in your application server and show how each of them is doing, things like no. of connections used,...


• In my environment i have only one data source /jdbc/sampleDB, select it and it will show usage graph for that connection along with the MVC wrapper information.


• Now select one of the Never Released messages and it will display the stack trace of the connection was never released. Did you notice that it is displaying the stack of calls that were made to get that connection. Now i know that the connection that i am getting in com.wpcertification.jdbc.ConnectionLeakServlet.doGet(ConnectionLeakServlet.java:41) is never released, so i can make changes to fix that code.