Run a Simple App with Spark

Steps

1. Install sbt

#

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 642AC823
sudo apt-get update
# the line below could be failed, I tried it several times
sudo apt-get install sbt

2. Prepare a simple project

as this one at my repo

3. Package the project

#

# go to the project folder, package it
# note it may failed several times
# saying cound not resolve dependency sbt 0.13.11
# for me it succeed after I viewed the error log and
# empty the error log file
# not sure whether this is a required step...
cd /home/benbai/Things/github/Spark/projects/SimplTesteApp
sbt package

4. run application by spark-submit

It will display lots of spark log, if you want to turn off the info logged by Spark you can add this log4j.properties into your SPARK_HOME/conf folder

#

$ YOUR_SPARK_HOME/bin/spark-submit \
--class "SimpleApp" \
--master local[4] \
target/scala-2.10/simple-project_2.10-1.0.jar 
##
##
##
##

Lines with a: 58, Lines with b: 26
##
##
##
##

References:


Installing sbt on Linux

Spark Quick Start

Install Spark 1.6.0 on Ubuntu 14.04

Environment:

Ubuntu 14.04 installed in VirtualBox

Steps:

1. Install JDK 8

###
# install oracle jdk 8
###
sudo apt-get -y install python-software-properties
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get -y update
sudo apt-get -y install oracle-java8-installer

# find out the path of your Java installation:
sudo update-alternatives --config java

# for me it returns
# There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-oracle/jre/bin/java

# edit /etc/environment
sudo gedit /etc/environment

# add this line into it, save and close
# JAVA_HOME="/usr/lib/jvm/java-8-oracle/jre/bin/java"
#
# then reload /etc/environment
source /etc/environment

2. Install scala 2.10

download scala 2.10.6 from http://www.scala-lang.org/download/2.10.6.html

extract it somewhere (I extract it to ~/Things/scala/scala-2.10.6)

again edit /etc/environment

add SCALA_HOME and PATH

after this the /etc/environment as below

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/home/benbai/Things/scala/scala-2.10.6/bin"
JAVA_HOME="/usr/lib/jvm/java-8-oracle/jre/bin/java"
SCALA_HOME="/home/benbai/Things/scala/scala-2.10.6"

reload /etc/environment and check scala

source /etc/environment
scala -version
# for me it returns
# Scala code runner version 2.10.6 -- Copyright 2002-2013, LAMP/EPFL

3. Get Spark 1.6.0

download spark from http://spark.apache.org/downloads.html

I choose 1.6.0 pre-build for hadoop 2.6 and later

extract it somewhere (I extract it to ~/Things/spark/spark-1.6.0-bin-hadoop2.6)

go to spark directory and test it

cd ~/Things/spark/spark-1.6.0-bin-hadoop2.6
./bin/spark-shell
# then test some command

Testing result:

References:

How To Install Java on Ubuntu with Apt-Get

Scala Download Page

Scala Getting Started

Spark Quick Start

Front-End Interview Questions and Answers

I have finished answering the Front-end-Developer-Interview-Questions, please refer to the answers at my git repo.

In my opinion, not only for interview, they really contains useful information/knowledge for Front-End/Full-Stack developers.

Hope this helps.

I've joined ZK Programmers Network at toptal dot com

As title, I was a developer/consultant of ZK at zkoss.org (2012 ~ 2015), I have joined ZK Programmers Network as a freelancer at toptal.com, hope I can help people who currently using ZK and make ZK more famous.

Honestly, it is because I have so much free time with current job so need to find something to do, currently I am answering the Front-end-Developer-Interview-Questions but almost done, so....

I am able to help with customize ZK Components, build new components,  build app/environment architecture, debug or performance tuning.

Grab me if needed, I believe I can give you a powerful hand :D