Tuesday, March 15, 2016

Run a Simple App with Spark


1. Install sbt


echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
# the line below could be failed, I tried it several times
sudo apt-get install sbt

2. Prepare a simple project

as this one at my repo

3. Package the project


# go to the project folder, package it
# note it may failed several times
# saying cound not resolve dependency sbt 0.13.11
# for me it succeed after I viewed the error log and
# empty the error log file
# not sure whether this is a required step...
cd /home/benbai/Things/github/Spark/projects/SimplTesteApp
sbt package

4. run application by spark-submit

It will display lots of spark log, if you want to turn off the info logged by Spark you can add this log4j.properties into your SPARK_HOME/conf folder


$ YOUR_SPARK_HOME/bin/spark-submit \
--class "SimpleApp" \
--master local[4] \

Lines with a: 58, Lines with b: 26


Installing sbt on Linux

Spark Quick Start


  1. Nice post . Thanks for the shearing valuable information. I really fiend this type blog. Special thanks to writer.
    Hire Angularjs Developer

  2. very nice blog, all the information is provided nicely and really a good site to read. I really admire your work, and thoughts, I also have a blog where I share the tutorials of programming to help the students to learn new programming skills, please have a look at it Four JavaScript examples for free and do suggest me to improve it.
    and keep posting the best articles, I am regular visitor of your site.
    Best Regards