Tuesday, March 15, 2016

Run a Simple App with Spark


1. Install sbt


echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
# the line below could be failed, I tried it several times
sudo apt-get install sbt

2. Prepare a simple project

as this one at my repo

3. Package the project


# go to the project folder, package it
# note it may failed several times
# saying cound not resolve dependency sbt 0.13.11
# for me it succeed after I viewed the error log and
# empty the error log file
# not sure whether this is a required step...
cd /home/benbai/Things/github/Spark/projects/SimplTesteApp
sbt package

4. run application by spark-submit

It will display lots of spark log, if you want to turn off the info logged by Spark you can add this log4j.properties into your SPARK_HOME/conf folder


$ YOUR_SPARK_HOME/bin/spark-submit \
--class "SimpleApp" \
--master local[4] \

Lines with a: 58, Lines with b: 26


Installing sbt on Linux

Spark Quick Start


  1. Nice post . Thanks for the shearing valuable information. I really fiend this type blog. Special thanks to writer.
    Hire Angularjs Developer

  2. very nice blog, all the information is provided nicely and really a good site to read. I really admire your work, and thoughts, I also have a blog where I share the tutorials of programming to help the students to learn new programming skills, please have a look at it Four JavaScript examples for free and do suggest me to improve it.
    and keep posting the best articles, I am regular visitor of your site.
    Best Regards

  3. Good blog posts information provided by the author
    Best Play and Pre School for kids in Hyderabad,India. To give your kid a best environment and learning it is the right way to join in play and pre school were kids can build there physically, emotionally and mentally skills developed. We provide programs to kids like Play Group, Nursery, Sanjary Junior, Sanjary Senior and Teacher training Program.
    Preschool in hyderabad

  4. Oleh karena itu, selalu percaya diri adalah langkah tepat untuk mendukung skill dalam menyusun kartu yang didapatkan.
    bandar ceme terpercaya
    paito warna terlengkap
    bocoran sgp

  5. Excellent explanation by the author
    Sanjary Academy is the best Piping Design institute in Hyderabad, Telangana. It is the best Piping design Course in India and we have offer professional Engineering Courses like Piping design Course, QA/QC Course, document controller course, Pressure Vessel Design Course, Welding Inspector Course, Quality Management Course and Safety Officer Course.
    Piping Design Course in Hyderabad ­

  6. Project Consultancy
    Anika Tech Support managed by a team of professionals utilising a combined 20+ years of experience in IT support and Managed IT services Focusing on Proactive support, continuity, growth and development. We are a forward thinking IT Company who support public and private sector using the latest tech as a driving force for our client’s Success.

  7. Nice blogging information provided

    Pressure Vessel Design Course is one of the courses offered by Sanjary Academy in Hyderabad. We have offer professional Engineering Course like Piping Design Course,QA / QC Course,document Controller course,pressure Vessel Design Course,Welding Inspector Course, Quality Management Course, #Safety officer course.
    Document Controller course
    Pressure Vessel Design Course
    Welding Inspector Course
    Safety officer course
    Quality Management Course
    Quality Management Course in India