How to use ElasticSearch from Web Application (How to fix java.lang.OutOfMemoryError: PermGen space error in Tomcat with ElasticSearch client as use case)

Recently i was working on a issue of Memory leak in the Web Application that talks to ElasticSearch. The problem was when i tried to deploy/undeploy the application few times without restarting server Tomcat ran out of memory and i got Exception in thread "http-bio-8080-exec-30" java.lang.OutOfMemoryError: PermGen space error. I wanted to figure out what is going on, so first i did create this simple web application that uses ElaticSearch client and provides a JAX-RS api to proxy calls to ElasticSearch.

	package com.csaa.aa.rest;
	import javax.ws.rs.Consumes;
	import javax.ws.rs.POST;
	import javax.ws.rs.Path;
	import javax.ws.rs.PathParam;
	import javax.ws.rs.Produces;
	import javax.ws.rs.core.MediaType;
	import org.elasticsearch.action.search.SearchResponse;
	import org.elasticsearch.client.Client;
	@Path("/search")
	public class SearchService {
	public SearchService() {
	super();
	System.out.println("Inside SearchService() constructor");
	}
	@POST
	@Path("/{index}/{type}")
	@Produces(MediaType.APPLICATION_JSON)
	@Consumes(MediaType.APPLICATION_JSON)
	public String search(@PathParam("index")String index, @PathParam("type") String type, String body){
	System.out.println("Entering SearchService.getResults()");
	System.out.println("Index " + index);
	System.out.println("Type " + type);
	System.out.println("Body " + body);
	Client client = ESClient.getClient();
	SearchResponse searchResponse = client.prepareSearch(index).setSource(body).execute().actionGet();
	return searchResponse.toString();
	}
	}

view raw SearchService.java hosted with ❤ by GitHub

This is how my sample ESClient.java looks like this takes care of establishing connection with ElasticSearch using TransportClient

	package com.webspherenotes.rest;
	import org.elasticsearch.client.Client;
	import org.elasticsearch.client.transport.TransportClient;
	import org.elasticsearch.common.settings.ImmutableSettings;
	import org.elasticsearch.common.settings.Settings;
	import org.elasticsearch.common.transport.InetSocketTransportAddress;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	public class ESClient {
	private static final Logger logger = LoggerFactory.getLogger(ESClient.class);
	private static TransportClient client;
	public static void startClient(){
	logger.debug("Entering ESClient.startClient()");
	Settings settings = ImmutableSettings
	.settingsBuilder().put("cluster.name", "csaaSearchDEV").build();
	InetSocketTransportAddress address = new InetSocketTransportAddress(
	"localhost", 8300);
	client = new TransportClient(settings);
	client.addTransportAddress(address);
	logger.debug("Exiting ESClient.startClient()");
	}
	public static Client getClient(){
	logger.debug("Entering ESClient.getClient()");
	if(client == null)
	startClient();
	logger.debug("Exiting ESClient.getClient()");
	return client;
	}
	}

view raw ESClient.java hosted with ❤ by GitHub

I could use the RESTClient to make a POST call for search like this

Now when i use the VisualVM for looking at the application, i could see elasticsearch created a threadpool of about 17 threads that are connected to the ElasticSearch like this

When i was stop/ my application for updating it, these threads created by ElasticSearch do not getting closed and because of the classes loaded by the web application cannot be garbage collected and i can see following warning messages on the console. [Dec 23, 2014 8:09:55 PM] [org.apache.catalina.loader.WebappClassLoader clearReferencesThreads] [SEVERE]: The web application [/ESClientWeb] appears to have started a thread named [elasticsearch[Blink][[timer]]] but has failed to stop it. This is very likely to create a memory leak. I see bunch of these messages in the console like this

When i use the Find Leak functionality on Apache Tomcat console i could see my application name shows up like this multiple times, once for every update

After couple of redeploy's my Tomcat runs out of memory and throws Exception in thread "http-bio-8080-exec-30" java.lang.OutOfMemoryError: PermGen space error like this

In order to solve this problem, all i had to do was to create ContextListener like this, i am creating object of org.elasticsearch.client.transport.TransportClient during startup and closing it during application stopping.

	package com.webspherenotes.rest;
	import javax.servlet.ServletContextEvent;
	import javax.servlet.ServletContextListener;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	public class ContextListener implements ServletContextListener {
	Logger logger = LoggerFactory.getLogger(ContextListener.class);
	public void contextInitialized(ServletContextEvent sce) {
	logger.debug("Entering ContextListener.contextInitialized()");
	ESClient.startClient();
	logger.debug("Exiting ContextListener.contextInitialized()");
	}
	public void contextDestroyed(ServletContextEvent sce) {
	logger.debug("Entering ContextListener.contextDestroyed()");
	ESClient.stopClient();
	logger.debug("Exiting ContextListener.contextDestroyed()");
	}
	}

view raw ContextListener.java hosted with ❤ by GitHub

I had to make couple of changes in ESClient.java like this

	import org.elasticsearch.client.Client;
	import org.elasticsearch.client.transport.TransportClient;
	import org.elasticsearch.common.settings.ImmutableSettings;
	import org.elasticsearch.common.settings.Settings;
	import org.elasticsearch.common.transport.InetSocketTransportAddress;
	import org.slf4j.Logger;
	import org.slf4j.LoggerFactory;
	public class ESClient {
	private static final Logger logger = LoggerFactory.getLogger(ESClient.class);
	private static TransportClient client;
	public static void startClient(){
	logger.debug("Entering ESClient.startClient()");
	Settings settings = ImmutableSettings
	.settingsBuilder().put("cluster.name", "csaaSearchDEV").build();
	InetSocketTransportAddress address = new InetSocketTransportAddress(
	"localhost", 8300);
	client = new TransportClient(settings);
	client.addTransportAddress(address);
	logger.debug("Exiting ESClient.startClient()");
	}
	public static void stopClient(){
	logger.debug("Entering ESClient.stop()");
	client.close();
	logger.debug("Exiting ESClient.stop()");
	}
	public static Client getClient(){
	logger.debug("Entering ESClient.getClient()");
	if(client == null)
	startClient();
	logger.debug("Exiting ESClient.getClient()");
	return client;
	}
	}

view raw ESClient.java hosted with ❤ by GitHub

After making these changes i could see the thread pool getting destroyed during shutdown of application and as a result the application classes are getting garbage collected and no more OutOfMemoryError