Installing Spark on MacOS High Sierra

Apache provides multiple ways to accomplish this depending on your personal preferences: use Homebrew, download a prebuilt file or build it yourself from source. This tutorial describes building it yourself from source. Apache’s source provides a build signature that installs all of your choice of prerequisites including: Maven, Scala, Hadoop, Yarn, and Zinc). So the only prerequisites you are responsible for are Maven 3.39+ or Java 8+.

Download the Java 8 MacOS dmg file for MacOS Sierra.
Install Maven.

Clone from git

Next we can clone the source for the build.

%cd /usr/local
%sudo git clone https://github.com/apache/spark.git

You may choose to sudo as yourself to build Spark but for later configuration you may also want to chown the folder so you can edit it

%sudo chown -R abe:admin /usr/local/spark

Building Spark with Maven

You can now build Spark with Yarn, Hadoop-2.7 and scala-2.11.

%cd /usr/local/spark
%export MAVEN_OPTS="-Xmx1300M -XX:MaxPermSize=512M -XX:ReservedCodeCacheSize=512m"
%./build/mvn -Pyarn -Phadoop-2.7 -Dscala-2.11 -DskipTests clean package

Once the build succeeds, edit your .bash_profile to include:

%export SPARK_HOME=/usr/local/spark
%export PATH=$PATH:$SPARK_HOME/bin

Then source your .bash_profile update:your settings.

%source .bash_profile

Test the Installation

Now you can test the installation.

%spark-shell