How to create Spark Scala fat jar with dependencies using Maven

If your developing a Spark application in Scala or your developing a Standalone Scala application and you want to create a fat jar that includes dependencies you can use following Maven Script as template for your build file. Couple of things are different here you must include scala-library as one of the library and then also include maven-scala-plugin that takes care of compiling the scala code. The value of sourceDirectory specifies the directory that contains your scala code. Then for packaging the project as a fat jar you can use maven-assembly-plugin as you would for any other java project built using Maven.

	<?xml version="1.0" encoding="UTF-8"?>
	<project xmlns="http://maven.apache.org/POM/4.0.0"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.spnotes.spark</groupId>
	<artifactId>HelloSparkStreaming</artifactId>
	<version>1.0-SNAPSHOT</version>
	<properties>
	<maven.compiler.source>1.7</maven.compiler.source>
	<maven.compiler.target>1.7</maven.compiler.target>
	<encoding>UTF-8</encoding>
	<scala.tools.version>2.11</scala.tools.version>
	<scala.version>2.11.0</scala.version>
	<spark.version>1.3.1</spark.version>
	</properties>
	<dependencies>
	<dependency>
	<groupId>org.scala-lang</groupId>
	<artifactId>scala-library</artifactId>
	<version>${scala.version}</version>
	<scope>provided</scope>
	</dependency>
	<dependency>
	<groupId>org.apache.spark</groupId>
	<artifactId>spark-core_${scala.tools.version}</artifactId>
	<version>${spark.version}</version>
	<scope>provided</scope>
	</dependency>
	<dependency>
	<groupId>org.apache.spark</groupId>
	<artifactId>spark-streaming_${scala.tools.version}</artifactId>
	<version>${spark.version}</version>
	<scope>provided</scope>
	</dependency>
	<dependency>
	<groupId>com.typesafe.scala-logging</groupId>
	<artifactId>scala-logging_${scala.tools.version}</artifactId>
	<version>3.1.0</version>
	</dependency>
	<dependency>
	<groupId>com.typesafe.scala-logging</groupId>
	<artifactId>scala-logging-slf4j_${scala.tools.version}</artifactId>
	<version>2.1.2</version>
	</dependency>
	</dependencies>
	<build>
	<sourceDirectory>src/main/scala</sourceDirectory>
	<plugins>
	<plugin>
	<groupId>org.scala-tools</groupId>
	<artifactId>maven-scala-plugin</artifactId>
	<version>2.15.2</version>
	<executions>
	<execution>
	<goals>
	<goal>compile</goal>
	</goals>
	</execution>
	</executions>
	</plugin>
	<plugin>
	<artifactId>maven-assembly-plugin</artifactId>
	<configuration>
	<archive>
	<manifest>
	<mainClass>com.spnotes.spark.NetcatStreamClient</mainClass>
	</manifest>
	</archive>
	<descriptorRefs>
	<descriptorRef>jar-with-dependencies</descriptorRef>
	</descriptorRefs>
	</configuration>
	</plugin>
	</plugins>
	</build>
	</project>

view raw pom.xml hosted with ❤ by GitHub