Run a Simple App with Spark

Steps

1. Install sbt

#

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
# the line below could be failed, I tried it several times
sudo apt-get install sbt

2. Prepare a simple project

as this one at my repo

3. Package the project

#

# go to the project folder, package it
# note it may failed several times
# saying cound not resolve dependency sbt 0.13.11
# for me it succeed after I viewed the error log and
# empty the error log file
# not sure whether this is a required step...
cd /home/benbai/Things/github/Spark/projects/SimplTesteApp
sbt package

4. run application by spark-submit

It will display lots of spark log, if you want to turn off the info logged by Spark you can add this log4j.properties into your SPARK_HOME/conf folder

#

$ YOUR_SPARK_HOME/bin/spark-submit \
--class "SimpleApp" \
--master local[4] \
target/scala-2.10/simple-project_2.10-1.0.jar 
##
##
##
##

Lines with a: 58, Lines with b: 26
##
##
##
##

References:


Installing sbt on Linux

Spark Quick Start