Invoking Python from Spark Scala project

When your developing your Spark code, you have option of developing it using either Scala, Java or Python. In some cases you might want to mix the languages that you want to use. I wanted to try that out so i built this simple Spark program that passes control to Python for performing transformation (All that it does it append word "python " in front of every line). You can download source code for sample project from here First thing that i did was to develop this simple python script that reads one line at a time from console, appends "Python " to the line and writes it back to standard console

	#!/usr/bin/env python
	import sys
	for line in sys.stdin:
	sys.stdout.write("Python "+line)

view raw echo.py hosted with ❤ by GitHub

Now this is how the driver looks like, most of the spark code is same only difference is lines.pipe("python echo.py") which says that pass every line in the RDD to python echo.py. and collect the output. Now there is nothing specific to python here, instead you could use any executable.

	package com.spnotes
	import org.apache.spark.{SparkContext, SparkConf}
	object ScalaPython {
	def main(argv:Array[String]): Unit ={
	System.out.println("Entering ScalaPython.main")
	if(argv.length != 1){
	println("Please provide 1 parameters <inputFileName>")
	System.exit(1)
	}
	val fileName = argv(0)
	val sparkConf = new SparkConf().setAppName("ScalaPython").setMaster("local")
	val sparkContext = new SparkContext(sparkConf)
	val lines = sparkContext.textFile(fileName)
	println("Print lines from original file")
	lines.foreach(println)
	val lines1 = lines.pipe("python echo.py")
	println("Print lines returned from python")
	lines1.foreach(println)
	System.out.println("Exiting ScalaPython.main")
	}
	}

view raw ScalaPython.scala hosted with ❤ by GitHub

When you run this code in cluster you should copy the python file on your machine say in spark directory then you can execute

bin/spark-submit 
    --files echo.py  
    ScalaPython-1.0-SNAPSHOT-jar-with-dependencies.jar helloworld.txt