Install Latest Apache Spark on Mac OS

Following is a detailed step by step process to install latest Apache Spark on Mac OS. We shall first install the dependencies : Java and Scala. To install these programming languages and framework, we take help of Homebrew and xcode-select.

This updated guide explains the current Homebrew method for installing Apache Spark on macOS, how to verify the installation, where Spark is installed, and how to fix common setup issues. The original command outputs are retained below for reference because they show what a successful installation session looks like, but the recommended commands for a new macOS setup are listed first.

Install Spark on Mac OS - Apache Spark Tutorial - www.tutorialkart.com
Install Spark on Mac OS

Apache Spark on macOS prerequisites before installation

For a clean Apache Spark installation on Mac, you need Terminal access, Homebrew, Apple Command Line Tools, and a compatible Java Development Kit. Apache Spark is not the same as the Spark Mail app for Mac; this tutorial is for the Apache Spark data processing engine used with Scala, Python, SQL, and Java workloads.

The Apache Spark downloads page lists the current stable Spark releases, and the Homebrew formula page shows the version that Homebrew will install through brew install apache-spark. If you need a specific Spark version for a project or cluster, check the official Apache Spark downloads page before installing.

  • macOS terminal tools: Install Apple Command Line Tools using xcode-select --install.
  • Package manager: Install or update Homebrew.
  • Java: Use a JDK version compatible with the Spark release you are installing.
  • Shell: Recent macOS versions use zsh by default, so persistent environment variables usually go in ~/.zshrc.

Recommended Homebrew commands to install Apache Spark on macOS

Use the following commands for a current Homebrew-based installation. These commands work for both Apple Silicon and Intel Macs because Homebrew resolves the correct package location for your machine.

</>
Copy
xcode-select --install
brew update
brew install openjdk
brew install apache-spark

If Java is already installed, check the active Java version before installing another JDK.

</>
Copy
java -version
/usr/libexec/java_home -V

After installing Spark, confirm the installed formula path with Homebrew. This is safer than hard-coding /usr/local or /opt/homebrew, because Intel Macs and Apple Silicon Macs commonly use different Homebrew prefixes.

</>
Copy
brew --prefix apache-spark
brew info apache-spark

Set SPARK_HOME and PATH for Apache Spark on macOS

Homebrew normally makes the Spark commands available automatically. If spark-shell or pyspark is not found, add Spark to your shell configuration. Use the Homebrew prefix command so the same setup works on both Apple Silicon and Intel Macs.

</>
Copy
echo 'export SPARK_HOME="$(brew --prefix apache-spark)/libexec"' >> ~/.zshrc
echo 'export PATH="$SPARK_HOME/bin:$PATH"' >> ~/.zshrc
source ~/.zshrc

If you use bash instead of zsh, add the same export lines to ~/.bash_profile or ~/.bashrc.

Verify Apache Spark installation on Mac using spark-shell

The simplest way to check whether Spark is installed on your Mac is to start the Spark shell and print the version. The shell may show warning messages about logging or native Hadoop libraries during local development; those warnings do not always mean the installation failed.

</>
Copy
spark-shell

At the scala> prompt, run:

</>
Copy
spark.version
spark.range(5).show()

A successful local check returns the Spark version and a small DataFrame output similar to this:

res0: String = 4.1.2
+---+
| id|
+---+
|  0|
|  1|
|  2|
|  3|
|  4|
+---+

Exit the Spark shell with:

</>
Copy
:quit

Check Apache Spark version on macOS without opening spark-shell

You can also check the Spark version directly from Terminal. This is useful when you only want to confirm the installation and do not need an interactive Scala shell.

</>
Copy
spark-submit --version

If Spark was installed by Homebrew, this command should print the Spark version, Scala version, Java version, and build details.

Run PySpark on macOS after installing Apache Spark

If you plan to use Apache Spark with Python, start PySpark after the Homebrew installation. Python support depends on your installed Spark release and Python version, so check the official PySpark installation documentation when you are setting up a project-specific environment.

</>
Copy
pyspark

At the PySpark prompt, run a small DataFrame command:

</>
Copy
spark.version
spark.range(5).show()

If you are using a virtual environment, install project dependencies inside that environment and keep the Spark installation path separate from your Python package environment.

Where Apache Spark is installed by Homebrew on Mac

Homebrew installs formulas under its prefix. On many Intel Macs, the prefix is /usr/local. On many Apple Silicon Macs, the prefix is /opt/homebrew. Do not assume one path; ask Homebrew for the installed path.

</>
Copy
brew --prefix
brew --prefix apache-spark
ls "$(brew --prefix apache-spark)/libexec"

The Spark home directory used by scripts is usually the libexec directory inside the Homebrew formula path.

Fix common Apache Spark macOS installation issues

spark-shell: command not found on macOS

If Terminal cannot find spark-shell, check whether the Homebrew binary path is available in your shell.

</>
Copy
brew --prefix apache-spark
echo $PATH
which spark-shell

If which spark-shell returns nothing, add Spark to PATH as shown in the SPARK_HOME section above, then restart Terminal.

Java gateway or unsupported class version error in Spark

Java errors usually mean the active JDK is not compatible with the Spark version being started. Check the active Java version and set JAVA_HOME to the JDK you want Spark to use.

</>
Copy
java -version
/usr/libexec/java_home -V

For example, if you want to use JDK 17 and it is installed on your Mac, you can set it for zsh like this:

</>
Copy
echo 'export JAVA_HOME=$(/usr/libexec/java_home -v 17)' >> ~/.zshrc
source ~/.zshrc

Native Hadoop library warning while starting Spark locally

When Spark starts on macOS, you may see a warning such as Unable to load native-hadoop library for your platform. For a local learning setup, Spark can still run with built-in Java classes. If you are connecting to a Hadoop cluster or using production Hadoop features, match your Spark, Hadoop, Java, and cluster versions carefully.

Original step-by-step Apache Spark Mac OS installation log

The following steps and outputs are from the original installation walkthrough. They are kept unchanged as a reference log, especially for readers comparing older Spark 2.x installations with current Homebrew installations.

Step 1 : Install Homebrew

Open Terminal. Run the following command in Terminal :

$ /usr/bin/ruby -e “$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)”

Enter the password if asked and continue.

==> Cleaning up /Library/Caches/Homebrew...
==> Migrating /Library/Caches/Homebrew to /Users/apple/Library/Caches/Homebrew..
==> Deleting /Library/Caches/Homebrew...
Already up-to-date.
==> Installation successful!
 
==> Homebrew has enabled anonymous aggregate user behaviour analytics.
Read the analytics documentation (and how to opt-out) here:
  http://docs.brew.sh/Analytics.html
 
==> Next steps:
- Run `brew help` to get started
- Further documentation: 
    http://docs.brew.sh

Step 2 : Install xcode-select

To install Java, Scala and Apache Spark through command line interface in Terminal, we shall install xcode-select. Enter and run the following command in Terminal :

$ xcode-select –install

Step 3 : Install Java

To install Java through command line, enter and run the following command in the Terminal :

$ brew cask install java
If your Java application still asks for JRE installation, you might need
to reboot or logout/login.
 
Installing this Cask means you have AGREED to the Oracle Binary Code
License Agreement for Java SE at
 
  https://www.oracle.com/technetwork/java/javase/terms/license/index.html
 
==> Satisfying dependencies
==> Downloading http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda
######################################################################## 100.0%
==> Verifying checksum for Cask java
==> Installing Cask java
==> Running installer for java; your password may be necessary.
==> Package installers may write to any location; options such as --appdir are i
Password:
==> installer: Package name is JDK 8 Update 144
==> installer: Upgrading at base path /
==> installer: The upgrade was successful.
? java was successfully installed!

Step 4 : Install Scala

To install Scala through command line, enter and run the following command in Terminal :

$ brew install scala
apples-MBP:~ Prasanth$ brew install scala
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
grafana                    idris                      passenger
 
==> Using the sandbox
==> Downloading https://downloads.lightbend.com/scala/2.12.3/scala-2.12.3.tgz
######################################################################## 100.0%
==> Downloading https://raw.githubusercontent.com/scala/scala-tool-support/0a217
######################################################################## 100.0%
==> Caveats
To use with IntelliJ, set the Scala home to:
  /usr/local/opt/scala/idea
 
Bash completion has been installed to:
  /usr/local/etc/bash_completion.d
==> Summary
? /usr/local/Cellar/scala/2.12.3: 44 files, 20.1MB, built in 1 minute 1 second

Step 5 : Install Spark

To install Apache Spark through command line, enter and run the following command in the Terminal :

$ brew install apache-spark
apples-MBP:~ Prasanth$ brew install apache-spark
==> Using the sandbox
==> Downloading https://www.apache.org/dyn/closer.lua?path=spark/spark-2.2.0/spa
==> Best Mirror http://www-eu.apache.org/dist/spark/spark-2.2.0/spark-2.2.0-bin-
######################################################################## 100.0%
? /usr/local/Cellar/apache-spark/2.2.0: 1,318 files, 221.5MB, built in 12 minutes 8 seconds

Step 6 : Verifying installation

To verify if the installation is successful, run the spark using the following command in Terminal :

$ spark-shell
apples-MBP:~ Prasanth$ spark-shell
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
17/08/01 21:52:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/01 21:52:58 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/08/01 21:52:58 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/08/01 21:52:59 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Spark context Web UI available at http://192.168.1.101:4040
Spark context available as 'sc' (master = local[*], app id = local-1501604572582).
Spark session available as 'spark'.
Welcome to
      __              __
     / _/_  _ __/ /_
    \ \/ _ \/ _ `/ _/  '_/
   /_/ ._/\,// //\\   version 2.2.0
      /_/
        
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_144)
Type in expressions to have them evaluated.
Type :help for more information.
 
scala>

We have successfully installed Apache Spark on Mac OS.

The installation directory would be /usr/local/Cellar/apache-spark/.

Apache Spark Mac installation FAQ

How do I know if Apache Spark is installed on my Mac?

Run spark-submit --version or open spark-shell. If Spark is installed correctly, Terminal prints the Spark version and starts the local Spark shell.

What is the latest Apache Spark version for macOS?

The latest version can change. Check the official Apache Spark downloads page and the Homebrew apache-spark formula page before installing. Homebrew installs the stable formula version available in its repository.

Do I need to install Scala separately for Apache Spark on Mac?

For a basic Homebrew installation, brew install apache-spark handles the formula dependencies. Install Scala separately only if you want the standalone Scala tools for development outside Spark.

Can I install Apache Spark on an Apple Silicon Mac?

Yes. Use Homebrew normally and avoid hard-coded Intel paths such as /usr/local. Use brew --prefix apache-spark to get the correct Spark path for your Mac.

Is Apache Spark for macOS the same as the Spark Mail app?

No. Apache Spark is a data processing engine. Spark Mail is an email client. This tutorial installs Apache Spark for local data processing and development.

Apache Spark on macOS editorial QA checklist

  • Confirm the current Spark version against the official Apache Spark downloads page before publishing a dated update.
  • Verify that all new command examples use current Homebrew syntax and not deprecated brew cask install java syntax.
  • Check that Apple Silicon and Intel Mac paths are explained through brew --prefix, not a single hard-coded path.
  • Test that spark-submit --version, spark-shell, and the small spark.range(5).show() example work on a clean macOS setup.
  • Keep the distinction clear between Apache Spark and unrelated Mac apps named Spark.

Conclusion

In this Apache Spark Tutorial, we have learnt to install latest Apache Spark on Mac OS . For a new macOS setup, the practical path is to install Homebrew and Apple Command Line Tools, install a compatible JDK, install Spark with brew install apache-spark, and verify the setup with spark-submit --version or spark-shell.