Apache Spark – Install on Ubuntu 16
Apache Spark can be run on majority of the Operating Systems. In this tutorial, we shall look into the process of installing Apache Spark on Ubuntu 16 which is a popular desktop flavor of Linux.
Java is the only dependency to be installed for Apache Spark.
To install Java, open a terminal and run the following command :
~$ sudo apt-get install default-jdk
Steps to Install latest Apache Spark on Ubuntu 16
1. Download Spark
There is a continuous development of Apache Spark. Newer versions roll out now and then. So, download latest Spark version when you are going to install.
To download latest Apache Spark release, open the url [http://spark.apache.org/downloads.html] in a browser.
2. Unzip and move spark to /usr/lib/
The download is a zip file. Before setting up Apache Spark in the PC, unzip the file. To unzip the download, open a terminal and run the tar command from the location of the zip file.
~$ tar xzvf spark-2.2.0-bin-hadoop2.7.tgz
Now move the folder to /usr/lib/ . In the following terminal commands, we copied the contents of the unzipped spark folder to a folder named spark. Then we moved the spark named folder to /usr/lib/.
~$ mv spark-2.2.0-bin-hadoop2.7/ spark ~$ sudo mv spark/ /usr/lib/
3. Add Path
Now we need to set SPARK_HOME environment variable and add it to the PATH. As a prerequisite, JAVA_HOME variable should also be set.
To set JAVA_HOME variable and add /usr/lib/spark/bin folder to PATH, open ~/.bashrc with any of the editor. We shall use nano editor here :
$ sudo nano ~/.bashrc
And add following lines at the end of ~/.bashrc file.
export JAVA_HOME=/usr/lib/jvm/default-java/jre export SPARK_HOME=/usr/lib/spark/bin export PATH=$PATH:SPARK_HOME
Latest Apache Spark is successfully installed in your Ubuntu 16.
4. Verify installation
Now that we have installed everything required and setup the PATH, we shall verify if Apache Spark has been installed correctly.
To verify the installation, close the Terminal already opened, and open a new Terminal again. Run the following command :
~$ spark-shell Using Sparks default log4j profile: org/apache/spark/log4j-defaults.properties Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/08/04 03:42:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 17/08/04 03:42:23 WARN Utils: Your hostname, arjun-VPCEH26EN resolves to a loopback address: 127.0.1.1; using 192.168.1.100 instead (on interface wlp7s0) 17/08/04 03:42:23 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address 17/08/04 03:42:36 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException Spark context Web UI available at http://192.168.1.100:4040 Spark context available as 'sc' (master = local[*], app id = local-1501798344680). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.2.0 /_/ Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_131) Type in expressions to have them evaluated. Type :help for more information. scala> :quit sparkuser@tutorialkart:~$
Also verify the versions of Spark, Java and Scala displayed during the start of spark-shell.
:quit command exits you from scala script of spark-shell.
In this Spark Tutorial, we have gone through a step by step process to make environment ready for Spark Installation, and the installation of Apache Spark itself.