WebSphere Portal Server Version 7.0 is available

The WebSphere Portal Server V 7.0 is now available for download. I am planning to download it from IBM's Partnerworld site. The documentation for the WPS 7.0 can be found here on the Portal Wiki. I am still looking for WPS 7.0 Infocenter

This is the WPS 7.0 Product documentation link that Dave Hay posted in the comment.

Adobe Flex as front end for J2EE application presentation

I did a presentation on how you can use Adobe Flex as front end of J2EE application Flex in Portal. This is good start if your a J2EE developer and want to learn what is flex, how you can integrate it in J2EE, what are your options,.. There are some slides which are specific to Portlet and WebSPhere Portal specifically

These are the sample applications that i built during preparation of this presentation

Client Side Optimization presentation

This weekend i went to attain ATech exchange in Austin. I did a presentation on Optimizing Web Site . This presentation about how you can optimize perceived performance of a web site.

Developing Adobe Flex chat client

Together the Adobe Flex framework and BlazeDS framework provide infrastructure to create messaging/chat type of application. You can use the infrastructure to either create one to one or broadcast type of application.

I wanted to try this feature so i built a simple broadcast application, in which there will be one chat destination and multiple clients and connect to it and publish and subscribe messages from it. You can download that application from here

First download the BlazeDS application and use it for creating a new J2EE application.

Then open the messaging-config.xml, which has messaging related configuration required for BlazeDS and add a destination element in it like this

<?xml version="1.0" encoding="UTF-8"?>
<service id="message-service" 
    class="flex.messaging.services.MessageService">

    <adapters>
        <adapter-definition id="actionscript" 
  class="flex.messaging.services.messaging.adapters.ActionScriptAdapter" 
  default="true" />
        <!-- <adapter-definition id="jms" 
  class="flex.messaging.services.messaging.adapters.JMSAdapter"/> -->
    </adapters>

    <default-channels>
        <channel ref="my-polling-amf"/>
    </default-channels>

 <destination id="chat"/>
</service>

The chat destination will be used by flex application for both publishing and subscribing messages

Next create a ChatClient.mxml file like this

<?xml version="1.0" encoding="utf-8"?>
<s:Application xmlns:fx="http://ns.adobe.com/mxml/2009" 
         xmlns:s="library://ns.adobe.com/flex/spark" 
         xmlns:mx="library://ns.adobe.com/flex/mx" minWidth="955" 
         minHeight="600" creationComplete="consumer.subscribe()">
  <fx:Declarations>

    <mx:Consumer id="consumer" destination="chat" message="messageHandler(event.message)"/>
    <mx:Producer id="producer" destination="chat"/>

  </fx:Declarations>
  <fx:Script>
    <![CDATA[
      import mx.messaging.messages.AsyncMessage;
      import mx.messaging.messages.IMessage;
      
      private function send():void{
        var message:IMessage = new AsyncMessage();
        message.body.chatMessage = msg.text;
        producer.send(message);
        msg.text = "";
      }
      private function messageHandler(message:IMessage):void{
        log.text += message.body.chatMessage + "\n";
      }
    ]]>
  </fx:Script>
  <mx:Panel title="Chat" width="100%" height="100%">
    <mx:TextArea id="log" width="100%" height="100%"/>
    <mx:ControlBar>
      <mx:TextInput id="msg" width="100%" enter="send()"/>
      <mx:Button label="Send B" click="send()"/>
    </mx:ControlBar>
  </mx:Panel>
</s:Application>

I am using declarative message handling, the mx:Consumer element declares that your interested in chat destination its messageHandler() method will get whenever there is a new message chat destination.

The mx:Producer element is used for publishing messages to the destination, i am calling its send() method to publish the text to the desination

Flex samples applications

These are sample Flex + portlet applications that you can download, most of the concepts are applicable to any J2EE application but there are some specific applications for WebSphere Portal Server

Potlet Specification compliant flex portlets

1. Flex HTTPService and JSON portlet


2. Flex HTTPService and XML portlet


3. Flex and WebServices Portlet


4. BlazeDS Contact Portlet


5. Developing Adobe Flex chat client


6. Making resource request from portlet using Flex application


7. Flex Youtube Demo portlet


8. Sample Flickr flex application

Flex portlets for Websphere Portal Server

1. Manipulating Portlet Preferences using Adobe Flex application


2. Working with user profile from flex application


3. How to make action or render request from the Adobe Flex


4. How to make action or render request from the Adobe Flex


5. Getting Edit or Help mode markup from the Flex portlet


6. Reading query string inside the Flex Portlet

Build scripts for flex applications

1. Maven Script to build flex application as part of J2EE


2. Ant Script to build flex application as part of J2EE

Why using mod_gzip might be counter productive with Internet explorer

The Apache mod_gzip module allows you to GZip traffic going through HTTP server on the fly. One problem with mod_gzip and mod_deflate is that they set Vary: Accept-Encoding header and you might be setting few other Vary headers.

Now problem with IE is that if it gets any response where value of Vary header is any thing other than User-Agent and Accept-Encoding then it does not cache the response and makes a full request every time.

These two articles have details on this problem

• Internet explorer meets Vary header


• Vary with care

What is the Vary HTTP header

The HTTP Vary response header lists all of the client request headers that the server considers to select the document or generate custom content.

For example if the server response depends on the User-Agent header, the Vary must include User-Agent. Ex. Lets say if your server side code can generate a version of document that does not use JavaScript for browsers that do not support JavaScript then you should add Vary: User-Agent header.

The Vary header is used by the caching proxies (Proxy could be your corporate proxy, ISP proxy, or some other proxy). Big portion of internet traffic goes through one or other proxy. The caching proxies store the response generated by server and when they get the next request for same resource, from either same client or different client they can return response from the cache instead of going to the originating server.

When a new request arrives the cache find the matching document. Before it can serve this document to the client it must see whether the server sent a Vary header in the cached response. If a Vary header is present, the header values for the header in the new request must match the header values in the old, cached request. Because server may vary their response based on the client request headers, caches must store both the client request headers and the corresponding server response headers witch each cached variant.

Include Style sheet at the top of the page

I never thought much about where and how to include css on my page but it seems that there is huge difference in how the page gets rendered if i include css at the top of the page instead of bottom of the page.

It also helps with problem called Flash of Unstyled Content (FOUC) , which means momentary flash of unstyled content is displayed to the user until CSS is downloaded

Good description of what happens when page is rendered in browser

This is a nice blog entry on what happens when is loaded in the browser and what type of trade offs browser make to show page as fast as possible

Using Selenium for testing portlets deployed in WebSphere Portal Server

I am big fan of unit testing, There are lots of advantages of good unit testing suite, one is its a very good way of documentation, it makes it easier to maintain the code and also it give you confidence even during development if you can run a big test suite on the code and everything works.

One problem with portlet development is its hard to develop unit test cases, because of lot of things, lets say if your using Portal API such as Controller SPI, or portlet model then how do you mock test that code or what if your code makes use of Ajax. Only option in that case would be to deploy your code in WPS and then test it through browser manually.

Recently i learned a tool called Selenium, which makes it easy to test the code running inside browser. The way it works is first you install a Firefox plug-in called Celenium IDE and use it to record user interaction inside the browser. Once the recording is done you can either play it back inside the browser or Selenium can create a JUnit test for you. The Junit test will actually lanuch the browser of your choice and play back the browser interaction you recorded. One good thing with Selenium is that it remembers the link name or form name that clicked on instead of the URLs so it works very well in the portlet world where urls are not constant or get generated on the fly.

I just figured out how to use Selenium to test the portlets deployed in WebSphere portal server and i thought i will share it with everyone. So first i created a simple SeleniumTestPortlet that shows a form to user in the view mode, when you enter user name and submit, the control will go to processAction(), inside the processAction i am reading the userName and setting it as render parameter and the user name is displayed in the generated HTMl after that. You can download the SeleniumTestPortlet from here. This is how the form looks like



Now i wanted to automate this part, so what i did is i used the Selenium IDE to test this interaction and verify that the userName is getting displayed in the output of doView() after i submit the form. After recording i converted the test in the Java- Junit Format which looks like this

package com.example.tests;

import com.thoughtworks.selenium.*;
import java.util.regex.Pattern;

public class Untitled 2 extends SeleneseTestCase {
 public void setUp() throws Exception {
  setUp("http://change-this-to-the-site-you-are-testing/", "*chrome");
 }
 public void testUntitled 2() throws Exception {
  selenium.open("/wpcert/demo/!ut/p/c4/04_SB8K8xLLM9MSSzPy8xBz9CP0os3hnd0cPE3MfAwMLT2cTA08TJ_8gI1d_AwNfQ_2CbEdFAGNKK9I!/");
  selenium.type("userID", "wasadmin");
  selenium.type("password", "wasadmin");
  selenium.click("ns_7_CGAH47L008IC40I4BOR2EO00E4__login");
  selenium.waitForPageToLoad("30000");
  selenium.click("link=fun");
  selenium.waitForPageToLoad("30000");
  selenium.click("link=Selenium Test Page");
  selenium.waitForPageToLoad("30000");
  selenium.type("userName", "Sunil Patil");
  selenium.click("submit");
  selenium.waitForPageToLoad("30000");
  verifyTrue(selenium.isTextPresent("User Name Sunil Patil"));
 }
}

After that i created a SeleniumTestCase as simple java project and added JUnit, Selenium-server.java and selenium-java-driver.jar to the build path. And i created a JUnit test case class like this

package com.webspherenotes.test;

import com.thoughtworks.selenium.*;
import java.util.regex.Pattern;

import org.openqa.selenium.server.SeleniumServer;

public class SecondTestCase extends SeleneseTestCase {
 
 private SeleniumServer seleniumServer;
 public void setUp() throws Exception {
  seleniumServer = new SeleniumServer();
  seleniumServer.start();
  setUp("http://localhost:10040/firefox", "*chrome");
 }
 public void tearDown() throws Exception {
  selenium.stop();
  seleniumServer.stop();
 }
 public void testCase() throws Exception {
  selenium.open("/wpcert/demo/!ut/p/c4/04_SB8K8xLLM9MSSzPy8xBz9CP0os3hnd0cPE3MfAwMLT2cTA08TJ_8gI1d_AwNfQ_2CbEdFAGNKK9I!/");
  selenium.type("userID", "wasadmin");
  selenium.type("password", "wasadmin");
  selenium.click("ns_7_CGAH47L008IC40I4BOR2EO00E4__login");
  selenium.waitForPageToLoad("30000");
  selenium.click("link=fun");
  selenium.waitForPageToLoad("30000");
  selenium.click("link=Selenium Test Page");
  selenium.waitForPageToLoad("30000");
  selenium.type("userName", "Sunil Patil");
  selenium.click("submit");
  selenium.waitForPageToLoad("30000");
  verifyTrue(selenium.isTextPresent("User Name Sunil Patil"));
 }
}

All that i had to do was add the lines shown in the bold to the code generated by Selenium IDE. Now i can run this code from my Eclipse IDE as unit test and i can see that a browser instance is opened and the same test is played back.

How to cache portal page in browser/proxy

WebSphere Portal allows you to configure a page so that it can be cached by user's browser or the proxy server. It does that by setting the appropriate values for Cache-control and Expires header.

I wanted to try this feature and see if i can configure a page to be cached for 1 day (i.e. 86400 seconds). In order to do that i had to configure WebSphere Portal so that it sets cache-control: max-age=86400. Also since i want the page to be cached across users, so this is how my cache-control header should look like Cache-Control public, max-age=86400

• First find out what all portlets are going to be displayed on the page that you want to cache, for each of those portlets set expirate-cache and public-scope header. In my case i have only one RemoteCachePortlet on the page so i configured it like this
  

  
  <?xml version="1.0" encoding="UTF-8"?>
  <portlet-app xmlns="http://java.sun.com/xml/ns/portlet/portlet-app_2_0.xsd"
    version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://java.sun.com/xml/ns/portlet/portlet-app_2_0.xsd 
    http://java.sun.com/xml/ns/portlet/portlet-app_2_0.xsd">
    <portlet>
      
      <portlet-name>RemoteCachePortlet</portlet-name>
      <display-name>Remote Cache Portlet</display-name>
      <portlet-class>com.webspherenotes.performance.RemoteCachePortlet</portlet-class>
      <expiration-cache>86400</expiration-cache>
      <cache-scope>public</cache-scope> 
      <supports>
        <mime-type>text/html</mime-type>
        <portlet-mode>view</portlet-mode>
      </supports>
      
      <portlet-info>
        <title>Remote Cache Portlet</title>
        <short-title>Remote Cache Portlet</short-title>
        <keywords>Remote Cache Portlet</keywords>
      </portlet-info>
  
    </portlet>
    
  </portlet-app>
  

  
  


• Next create a page and configure the Page Cache Options like this
  
  


• Last step would be to add two set these two properties at the Navigation Service like this
  
  

Once the page is configured try accessing the page and you should see the headers being set like this




You can see that Cache-control header is being set and now if you look for the cache-entry in the browser you will notice that page is supposed to get expired in 24 hrs from the time you accessed it




Important Note The headers returned by portal server look ok but when i tried it on local somehow the IE browser is making request to the server. But the Google's Chrome browser is not it is able to deal with cache properly. Not sure if something else is causing this

Navigator Service

The WebSphere Portal Server has Navigator Service that allows you to configure how your portal page (By that i mean HTML generated by portal) gets cached in users browser as well as proxy server. It lets you configure value of Cache-control and Expires header.

# Cache expiration time (in seconds) for the portal internal cache for the 
# unpersonalized and unauthenticated portal page.
# (values <= 0 are automatically replaced with the default)


# Default: 60
#public.reload = 60


# The cache expiration time (in seconds) for caches outside of portal server
# and for unauthenticated pages only.
# These caches must adhere to the HTTP 1.1 specification (RFC 2616)
# and specifies the time when HTTP caches should drop the response.
#
# This value will be used as a maximum value for the cache expiry time and
# as a global default value for unauthenticated pages. 
# In case the setting remote.cache.expiration is also set to a value 
# greater than or equal to 0 the minimum value of both settings will be used.
#
# Note that portal may reduce the cache lifetime or may reduce the cache scope
# (public, private) or may switch off the overall cacheability of page
# while processing a request. 
# So this value might not be static for all responses to requests 
# to unauthenticated pages. 
#
# The response of portal server will set the following header fields:
# - The 'Expires' header with the expiry time added to the 
#   system date & time of the portal server 
# - The 'Cache-Control : max-age =' header with the expiry time as its parameter

# Default: 60
#public.expires = 60


# The remote cache expiration (in seconds) for caches outside of portal server
# and for authenticated as well as for unauthenticated pages 
# These caches must adhere to the HTTP 1.1 specification (RFC 2616)
# and specifies the time when HTTP caches should drop the response.
#
# This value will be used as a maximum value for the cache expiry time 
# for unauthenticated and authenticated pages.
# The global default value for authenticated pages will be 0.
# In case the setting public.expires is also set to a value 
# greater than or equal to 0 the minimum value of both settings will be used
# as the global default value for unauthenticated pages.
#
# Note that portal may reduce the cache lifetime or may reduce the cache scope
# (public, private) or may switch off the overall cacheability of page
# while processing a request. 
# So this value might not be static for all responses to requests 
# to unauthenticated and authenticated pages. 
#
# The response of portal server will set the following header fields:
# - The 'Expires' header with the expiry time added to the 
#   system date & time of the portal server 
# - The 'Cache-Control : max-age =' header with the expiry time as its parameter

# Default for full page responses (in seconds)
#remote.cache.expiration = 10800


# The remote cache information for fragment responses (i.e. responses which do
# not refresh the entire page but single page parts only, e.g. a single portlet).
# The specified value will be used as a maximum value for the cache expiry time
# (i.e. a fragment response will never be cacheable longer than the specified value
# even if the fragment itself specifies a value that is greater than the threshold
# specified here).
#
# If an addressed fragment does not specify an expiry value the portal server will
# use the default 0 i.e. the response will not be cacheable.
#
# Special values:
# - Specify a value of -1 to indicate that fragment responses should not underlie a
#   a certain treshold.
# - Specify a value of 0 to prevent fragment caching.   

# Default for fragment responses (in seconds)
#remote.cache.expiration.fragment = -1


# The default remote cache information for model feeds (typically ATOM responses
# carrying model data. The supported models are content model (identifier "cm"),
# navigation model (identifier "nm"), layout model (identifier "lm"), and portlet
# model (identifier "pm"). If no value is provided the feed expiry will default to 0.

# Default for feeds (in seconds)
#remote.cache.expiration.feed.cm = 60
#remote.cache.expiration.feed.nm = 60
#remote.cache.expiration.feed.lm = 60
#remote.cache.expiration.feed.pm = 60


# The vary response header indicates the set of request-header fields 
# that fully determines whether a cache outside of portal is permitted 
# to use a response to reply to a subsequent request.
# These caches must adhere to the HTTP 1.1 specification (RFC 2616).
#
# The response of portal server will set the following header fields:
# - The 'Vary' header with the specified parameter
# If more than one request-header field shall be referenced as the parameter value
# it must be provided as comma separated list like the original vary header
# as specified in the HTTP 1.1 specification. 

# Default for full page responses
#vary = User-Agent, Cookie


# The vary response header for fragment responses (i.e. responses which do not
# refresh the entire page but single page parts only, e.g. a single portlet).

# Default for fragment responses
#vary.fragment = User-Agent


# Specifies whether the unauthenticated (anonymous) user always has a session.
# This may be desirable because a portlet requires a session for
# anonymous users. However, remember that having a session forbids
# any HTTP proxy to cache the response.

# Default: false
#public.session = false

Remote portal page caching

If you use WebSphere portal for displaying pages to anonymous user and the same page can be shared across multiple users then you can use remote caching, which means the HTML generated by the WebSphere Portal will be cached by users browsers as well as proxy and

If you use a proxy server such as WebSphere Edge Server and your system has content with anonymous access rights that can be shared among multiple users, you can improve performance by caching this shared content

Trimming white space and entra lines from the markup generated by WebSphere Application Serve

If you look at the HTML markup generated for IBM's default page you will notice that it has lot of white spaces and though they increase size of the response they are not required to display the page properly. If you set the com.ibm.wsspi.jsp.usecdatatrim custom property to true for your JSP engine, all of the white space and extra lines in the generated Java code are stripped out.

Take a look at the page which has two portlets and default IBM theme,



The page that gets generated is 64kb's in size and if you look at the view source you can see all the white spaces.

Now set com.ibm.wsspi.jsp.usecdatatrim equal to true as Web Container Custom property like this



Restart the server and try accessing the same page, you will notice that the size of the markup is reduced to 56.4 kb (Which is saving of close to 15 %) like this



The saving size could be much larger if your page is more complex.

Setting Web Container Custom properties

The WebSphere Application Server allows you to configure behavior of the Web Container at system level by setting some predefined custom properties.

You can set custom Web Container properties using WAS Admin Console, by going to Application servers > servername > Web container > Custom Properties. This is screen shot of custom properties that are set on my local WebSphere Portal Installation



These are list of properties and their meanings in WAS 6.1

1. com.ibm.ws.webcontainer.disallowserveservletsbyclassname: When the serveServletsByClassnameEnabled property is enabled, it is possible to access servlets directly, resulting in a possible security exposure. Define the following custom property to disallow the use of the serveServletsByClassnameEnabled property across the entire application server level.


2. com.ibm.ws.webcontainer.donotservebyclassname: A semi-colon delimited list of classes to be completely disallowed from being served by class name.


3. com.ibm.ws.webcontainer.suppressheadersinrequest: The com.ibm.ws.webcontainer.suppressheadersinrequest custom property can be used to suppress the inclusion of request headers that start with special characters, such as "$" or "_" . Some applications do not handle request headers that start with special characters


4. com.ibm.ws.webcontainer.webgroupvhostnotfound: Error message SRVE0017W states "Web Group not found: {0}", and error message SRVE0255 states "A WebGroup/Virtual Host to handle {0} has not been defined". These messages might be returned when the application that is called to process the request serviced by IBM WebSphere Application Server is not found. You can use the com.ibm.ws.webcontainer.webgroupvhostnotfound custom property to change the text of these message to text that is more suitable for your environment.


5. DecodeUrlAsUTF8: The UTF-8 encoded URL feature, which provides UTF-8 encoded Uniform Resource Locators (URLs) to support the double-byte characters in URLs is enabled by default. You can prevent the Web container from explicitly decoding URLs in UTF-8 and have them use the ISO-8859 standard as per the current HTTP specification by setting value of DecodeUrlAsUTF8 to false


6. listeners: The servlet specification supports applications registering listeners for servlet-related events on an individual application basis through the web.xml descriptor. However, using the listeners custom property, a server can listen to servlet events across Web applications. To implement global listening, a listener is registered at the Web container level and is propagated to all of the installed and new Web applications. You can create a Servlet Listener following the J2EE specification and specify its name here to apply it to all the web applications


7. DebugSessionCrossover:The DebugSessionCrossover custom property enables code to perform additional checks to verify that only the session associated with the request is accessed or referenced. Messages are logged if any discrepancies are detected.


8. com.ibm.wsspi.jsp.disableTldSearch: The com.ibm.wsspi.jsp.disableTldSearch custom property can be used to improve application startup time. By default, when an application starts, the JSP engine searches the application installation directories for the taglib descriptor (TLD) files. This search process might increase the startup time for large applications with a large number of files and directories. To disable this search process, set the value of this property to true


9. com.ibm.ws.webcontainer.disallowAllFileServing:You can enable file serving on a global level across a given application server by using the fileServingEnabled custom property. However, the fileServingEnabled property is overridden by the specific deployment information of each application. Therefore,, the current fileServingEnabled custom property only applies as a backup in case an application does not define the fileServingEnabled setting itself. To globally override this setting on a specific application server to prevent the application server from serving static files regardless of their individual deployment settings, set the Web container custom property com.ibm.ws.webcontainer.disallowAllFileServing to true.


10. com.ibm.ws.webcontainer.mapFiltersToAsterisk:When processing a request, the Web container recognizes servlet mappings to "*" as the same as servlet mappings to "/*". To provide the same behavior with filter mapping, set the com.ibm.ws.webcontainer.mapFiltersToAsterisk custom property to true. Setting the com.ibm.ws.webcontainer.mapFiltersToAsterisk custom property to true causes the Web container to recognize filter mappings to "*" as a filter mapping to "/*" . This custom property is not case sensitive.

Client Side Aggregation Theme

I did some research on how the Client Side Aggregation (CSA) theme works and this presentation has my notes about that

Client Side Aggregation

Ant Task to compress JavaScript and CSS using YUICompressor

In the Ant task for combining multiple JavaScript files into one and minifying them using Dojo Shrinksafe entry i built a sample ant task that can be used to combine and minify JavaScript using Dojo ShrinkSafe.

But what if you want to use the YUI Compressor, or you want to combine and minify css, so i built this task for that

<target name="minify" >
<antcall target="minifyjs" />
<antcall target="minifycss" />
</target>
<target name="minifyjs" >
<mkdir dir="build/WebContent/js" />
<concat destfile="build/WebContent/js/combined.js">
 <fileset dir="WebContent/js" includes="*.js" />
</concat>
<java fork="true" jar="${yuicompressor.lib.dir}" dir="build/WebContent/js" 
output="build/WebContent/js/combined.min.js">
  <arg value="combined.js" />
</java>  
</target>

<target name="minifycss" >
<mkdir dir="build/WebContent/css" />
<concat destfile="build/WebContent/css/combined.css">
 <fileset dir="WebContent/css" includes="*.css" />
</concat>
<java fork="true" jar="${yuicompressor.lib.dir}" dir="build/WebContent/css" 
output="build/WebContent/css/combined.min.css">
  <arg value="combined.css" />
</java>  
</target>

The minify task is broken into two parts one is minifyjs which combines and minfies all the JavaScript files in /js folder and the minifycss which combines and minifies all the css files in /css folder.

Compressing the JavaScript and CSS files on the fly using YUI COmpressor

I was trying to build a Servlet and Filter which can be used to minify JavaScript and CSS on the fly using YUI COmpressor. But i ran into this exception. When i was searching for solution to this problem i found WebUtils project which has
following components which can use easily used in J2EE webapp.

• JSCSSMergeServlet - to merge multiple JS or CSS and serve them in one HTTP request.


• YUIMinFilter - to compress JS or CSS resources on the fly using YUI compressor.


• YuiMinTag - Custom tag to compress inline JS and CSS code using YUI compressor in JSPs.

[8/9/10 11:46:59:322 PDT] 0000008d ServletWrappe E   SRVE0068E: Uncaught exception thrown in one of the service methods of the servlet: YUICompressorDemoServlet. Exception thrown : java.lang.StringIndexOutOfBoundsException
        at java.lang.String.substring(String.java:1088)
        at com.yahoo.platform.yui.compressor.JavaScriptCompressor.printSourceString(JavaScriptCompressor.java:268)
        at com.yahoo.platform.yui.compressor.JavaScriptCompressor.parse(JavaScriptCompressor.java:333)
        at com.yahoo.platform.yui.compressor.JavaScriptCompressor.(JavaScriptCompressor.java:536)
        at com.webspherenotes.performance.minfy.YUICompressorDemoServlet.doGet(Unknown Source)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:743)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:856)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1146)
        at com.ibm.ws.webcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:592)
        at com.ibm.ws.wswebcontainer.servlet.ServletWrapper.handleRequest(ServletWrapper.java:524)
        at com.ibm.ws.webcontainer.webapp.WebApp.handleRequest(WebApp.java:3517)
        at com.ibm.ws.webcontainer.webapp.WebGroup.handleRequest(WebGroup.java:269)
        at com.ibm.ws.webcontainer.WebContainer.handleRequest(WebContainer.java:818)
        at com.ibm.ws.wswebcontainer.WebContainer.handleRequest(WebContainer.java:1478)
        at com.ibm.ws.webcontainer.channel.WCChannelLink.ready(WCChannelLink.java:125)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleDiscrimination(HttpInboundLink.java:458)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.handleNewInformation(HttpInboundLink.java:387)
        at com.ibm.ws.http.channel.inbound.impl.HttpInboundLink.ready(HttpInboundLink.java:267)
        at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.sendToDiscriminators(NewConnectionInitialReadCallback.java:214)
        at com.ibm.ws.tcp.channel.impl.NewConnectionInitialReadCallback.complete(NewConnectionInitialReadCallback.java:113)
        at com.ibm.ws.tcp.channel.impl.AioReadCompletionListener.futureCompleted(AioReadCompletionListener.java:165)
        at com.ibm.io.async.AbstractAsyncFuture.invokeCallback(AbstractAsyncFuture.java:217)
        at com.ibm.io.async.AsyncChannelFuture.fireCompletionActions(AsyncChannelFuture.java:161)
        at com.ibm.io.async.AsyncFuture.completed(AsyncFuture.java:136)
        at com.ibm.io.async.ResultHandler.complete(ResultHandler.java:196)
        at com.ibm.io.async.ResultHandler.runEventProcessingLoop(ResultHandler.java:751)
        at com.ibm.io.async.ResultHandler$2.run(ResultHandler.java:881)
        at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1497)

Ant task for combining multiple JavaScript files into one and minifying them using Dojo Shrinksafe

In the Combining and minifying multiple JavaScript files into one on the fly, entry i built a Servlet that can be used to combine multiple JavaScript files into one. But its always better to combine multiple JavaScript into one during the build time instead of run time.

So this is the Ant task that you can add to your build file which can do the combining and minfiying for you

<target name="shrinksafe" >
    <concat destfile="WebContent/js/combined.js">
        <fileset dir="WebContent/js" includes="*.js" />
    </concat>
    <java fork="true" jar="${shrinksafe.lib.dir}/shrinksafe.jar" 
 dir="build/WebContent/js" 
 output="WebContent/js/combined.min.js">
       <classpath>
         <pathelement location="${shrinksafe.lib.dir}/js.jar"/>
      </classpath>
      <arg value="combined.js" />
    </java>  
  </target>

The task can be broken into two pieces first you will have to combine multiple JavaScript files into one and i am using concat element for it, what it does is find all the JavaScript files inside WebContent/js and combine them together into combined.js which will be saved in WebContent/js folder.

Once the combined JavaScript file is ready i am executing the shrinksafe.jar, passing it the combined.js as argument and the output of this command would be saved into combined.min.js. Hereafter whenever you want to include the combined javascript include combined.min.js on your page.

Combining and minifying multiple javascript files into one on the fly

In the On the fly JavaScript minification using Dojo ShrinkSafe entry i built a simple ServletFilter that can be used to minify JavaScript files on the fly.

After i published that entry i got email from a reader that how can we combine and minify multiple JavaScript file into one so i built this filter which can be used to minfiy multiple JavaScript files into one and then minify the resultant file using Dojo Shrinksafe

package com.webspherenotes.performance.minfy;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.StringTokenizer;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.dojotoolkit.shrinksafe.Compressor;

/**
 * Servlet implementation class ShrinkSafeAntDemoServlet
 */
public class ShrinkSafeAntDemoServlet extends HttpServlet {
  private static final long serialVersionUID = 1L;

    /**
     * Default constructor. 
     */
    public ShrinkSafeAntDemoServlet() {
        // TODO Auto-generated constructor stub
    }

  /**
   * @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
   */
  protected void doGet(HttpServletRequest request, HttpServletResponse response) 
  throws ServletException, IOException {
    response.setContentType("application/javascript");
    ArrayList fileList = getFileList(request.getParameter("fileList"));
    String combinedFile = getCombinedFile(fileList);
    System.out.println("Combined JavaScript " + combinedFile);
    String outputJavaScript = Compressor.compressScript(combinedFile, 0, 1, "all");
    response.getWriter().println(outputJavaScript);
  }
  
  private ArrayList getFileList(String fileListStr){
    StringTokenizer st = new StringTokenizer(fileListStr, ",");
    ArrayList fileList = new ArrayList();
    while(st.hasMoreElements()){
      fileList.add((String)st.nextElement());
    }
    return fileList;
  }
  
  private String getCombinedFile(ArrayList fileList) throws IOException{
    StringBuffer combinedJavaScript = new StringBuffer();
    for(int i = 0 ; i < fileList.size(); i++){
      String  fileName = (String)fileList.get(i);
      System.out.println("Read the file " + fileName);
      BufferedReader br = new BufferedReader( new InputStreamReader( 
   getServletContext().getResourceAsStream(fileName)));
      String line = null;
      while((line =br.readLine()) != null){
        combinedJavaScript.append(line);
        combinedJavaScript.append("\n");
      }
    }
    return combinedJavaScript.toString();
  }
}

You can call this servlet passing it list of JavaScript files that you want to combine as argument, i am assuming that those files are part of the same web application as that of the servlet. Assume that you have test.js, test1.js and test2.js files in your web application and instead of including each one of them separately on a page you want to include all 3 of them together like this.

< script 
src='/shrinksafe/ShrinkSafeAntDemoServlet?fileList=/js/test.js,/js/test1.js,/js/test2.js' >

The ShrinkSafeAntDemoServlet first reads value of fileList parameter and parse it to find out the number of files that you want to combine, then getCombinedFile() method takes list of JavaScript files as argument and reads and combines them into a String. Once the String is read it passes control to Compressor.compressScript() method of Dojo compressor to minify the mulitple JavaScript files

Why you should not use ETag

In the Last-modified time stamp validation entry i mentioned following problems with using Last-Modified date

• A file's time stamp might get updated without any changes in the actual content of the file. In that case any conditional get request will result in 200 response and will send the full body of resource


• One of the common problems in that HTTP servers clocks are out of synch. Even if your environment has multiple HTTP servers they might not have same time. So if you copy same file to different server at the same time, it might end up getting different last-modified time. SO if your request goes to different HTTP server, the last modified time wont match and it will return 200 for file that is not changed


• If-modified-since values cannot be used for objects that may be updated more frequently than once per second, because value of Last-Modified is specified in seconds

So if your thinking that you could use ETag to solve this problem but problem is that the Http Server generates Last-Modified date by default and the browser always adds If-Modified-Since header in the subsequent request. As per HTTP 1.1 specification client must use an entity tag validator if a server sends back an entity tag. If the server sends only a Last-Modified value, the client can use If-Modified-Since validator. If both an entity tag and last-modified date are available, the client should use both re-validation schemes. If an HTTP 1.1 cache or server receives a request with both If-Modified-Since and entity tag conditional headers, it must not return a 304 Not Modified response unless doing so is consistent with all of the conditional header fields in the request.

That means for the response to be valid and returning 304(which is desirable behavior) both Last-Modified and ETag must be same as that of the copy on the server. That means if the Last-Modified date does not match for any reason but ETag matches server will still return 200 status code with full response. But if your using default format of ETag INode MTime Size then there is chance that the server might generate different ETag for same file, which will result in full refresh even if the client has latest version.

The Configure ETag entry on YSlow blog has details on why you should disable ETag

How to enable ETag in Apache Http Server

If your using Apache HTTP Server then you can use it to generate and return ETag for static resources that are served from disk. The Apache Http Server also takes care of comparing the value of If-None-Match header with ETag of the resource and returning either 304 Not modified or 200 OK.

Support for ETag is part of the Apache Core and it is enabled by default. Ex. These are the response headers that i get when i try to access cachesample.gif, which is a static image file on disk from Apache Server



The value of ETag header in this case is combination of three things INode MTime Size

You can configure the behavior of the ETag using the FileETag directive which configures the file attributes that are used to create the ETag (entity tag) response header field when the document is based on a file. (The ETag value is used in cache management to save network bandwidth.) In Apache 1.3.22 and earlier, the ETag value was always formed from the file's inode, size, and last-modified time (mtime). The FileETag directive allows you to choose which of these -- if any -- should be used. The recognized keywords are:

1. INode : The file's i-node number will be included in the calculation. i-node is the number generated by OS to keep track of the file, it includes things like access level, creation time,... You can configure apache to use only INode by adding this line to httpd.conf
   
FileETag INode

   This is screen of how the INode only ETag looks like
   
   


2. MTime: The date and time the file was last modified will be included. You can configure apache to use only Last-Modified date of the file to generate ETag by adding this line to httpd.conf
   
FileETag MTime

   This is how my cachecontrol.gif response headers look like when i use MTime for ETag generation
   
   


3. Size: The number of bytes in the file will be included. You can configure Apache to use only size of the file for generating ETag by adding this line to httpd.conf
   
FileETag Size

   This screen shot of headers when Apache is configured to use only file Size for calculating ETag
   
   


4. All: All available fields will be used. This is equivalent to: FileETag INode MTime Size


5. None: You can disable generation of ETag by Apache Http Server by adding this line
   
FileETag None

   THis is screen shot of headers after ETag is disabled
   
   

Using Entity Tag (ETag) for validation

In the Last-modified time stamp validation, i talked about how you can use Last-Modified date for making conditional request and problems with that approach.

The Http 1.1 specification provides another kind of validator known as an entity tag(ETag) . An entity tag is nothing but a string that is used to identify a specific instance of an object.

When you request a resource, server can calculate string representing the version of the resource and return it to the client using ETag header like this


Etag "9c334-9933-74b9cec0"


After that whenever browser wants to check if it has the correct version of the resource it will add following header to the conditional request


If-None-Match "9c334-9933-74b9cec0"


Server will check the version in If-None-Match the version of resource that it has and will return either 304 (Not modified) if the version is same or 200 with full response body if the resource is changed.

Important Note: As per HTTP 1.1 specification client must use an entity tag validator if a server sends back an entity tag. If the server sends only a Last-Modified value, the client can use If-Modified-Since validator. If both an entity tag and last-modified date are available, the client should use both re-validation schemes. If an HTTP 1.1 cache or server receives a request with both If-Modified-Since and entity tag conditional headers, it must not return a 304 Not Modified response unless doing so is consistent with all of the conditional header fields in the request.

Last-modified time stamp validation

In the What is conditional/ validation request i explained concept of conditional request, which means the server will execute the request only if particular condition is met.

When a browser requests any resource, sometime the server response includes Last-modified header that specifies the time when the resource was last changed on the origin server like this


Last-Modified Thu, 22 Jul 2010 23:41:23 GMT


This header tells the client that the requested resource was last modified on 22nd of July 2010. If requested resource is static file served by the HTTP Server, then this value would be equal to the file system modification time. The Last-modified time stamp is given in the Greenwich Mean Time (GMT) with one second resolution.

When the browser/cache wants to validate if the resource that it has is changed, it will take the value of Last-modified header from the response that it already has and make a conditional get request by adding If-Modified-Since header to the request like this


If-Modified-Since Thu, 22 Jul 2010 23:41:23 GMT


These are the disadvantages of using If-Modified-Since header for making conditional request

• A file's time stamp might get updated without any changes in the actual content of the file. In that case any conditional get request will result in 200 response and will send the full body of resource


• One of the common problems in that HTTP servers clocks are out of synch. Even if your environment has multiple HTTP servers they might not have same time. So if you copy same file to different server at the same time, it might end up getting different last-modified time. SO if your request goes to different HTTP server, the last modified time wont match and it will return 200 for file that is not changed


• If-modified-since values cannot be used for objects that may be updated more frequently than once per second, because value of Last-Modified is specified in seconds

What is conditional/ validation request

The HTTP Specification has concept of conditional/ validation request, which client makes to check if the cached copy that it has is still valid. Client will make the conditional get request in two cases

• Resource is not cacheable: If it made a request for resource and got the response, but request does not have either Expires or cache-control header or you explicitly set the resource as non-cachable by setting Expires equal to 0 or in the past date, or set Cache-control: no-cache.


• Cached resource is expired: A resource is cachable and lets say it was cacheable till 1 PM on 1st of August 2010, then if you query for the resource at 1.30 pm, in that case browser/ cache will make a request

When a browser makes conditional request either of two things will happen

• Client copy is fresh: That means the copy that browser has is same as that of the copy of the server or the resource is not modified. In that case the server sends HTTP status code 304 (Not modified) with only headers without body. The server can send new expires date for the resource so that the resource can get cached in the browser


• Client copy is stale: Means the server has new copy of the resource, or the resource has changed since the last time user requested for it. In that case the server will return HTTP status code 200 OK, with full resource in the body.

Conditional requests are implemented by conditional headers that start with If. The conditional header allows method to execute only if particular conditional is met

GET /perf/images/cachesample.gif HTTP/1.0
If-Modified-Since Thu, 22 Jul 2010 23:41:23 GMT


Means return /perf/images/cachesample.gif, only if it was modified since 22nd of July

There are two different attributes that you can use for testing creating conditional request

• Last Modified date: Means check if the document has changed since the last modified date


• ETag: Used to check if the entity tag(ETag) of the document has changed