[Spark] Spark installation on osx
ITWeb/개발일반 2017. 11. 18. 21:54기본 설치 입니다.
제가 brew install 보다는 직접 binary 받아서 설치 하는걸 더 선호해서 올려 봅니다.
[Spark installation on osx]
Ref. https://isaacchanghau.github.io/2017/06/28/Spark-Installation-on-Mac-OS-X/
1. scala
https://www.scala-lang.org/download/
$ tar -xvzf scala-2.12.4.tgz
$ vi .bash_profile
export SCALA_HOME=/Users/henry/Work/apps/scala-2.12.4
PATH=$SCALA_HOME/bin:$PATH
2. spark
https://spark.apache.org/downloads.html
$ tar -xvzf spark-2.2.0-bin-hadoop2.7.tgz
$ vi .bash_profile
export SPARK_HOME=/Users/henry/Work/apps/spark-2.2.0-bin-hadoop2.7
PATH=$SPARK_HOME/bin:$PATH
$ cd /Users/henry/Work/apps/spark-2.2.0-bin-hadoop2.7
$ cp spark-env.sh.template spark-env.sh
$ vim spark-env.sh
export SCALA_HOME=/Users/henry/Work/apps/scala-2.12.4
export SPARK_MASTER_IP=localhost
export SPARK_WORKER_MEMORY=1g
$ spark-shell
scala 와 spark 이 설치가 되어야 합니다.
보시면 아시겠지만 .bash_profile 에 환경 설정 해주시고 실행하시면 됩니다.
PATH=$SCALA_HOME/bin:$SPARK_HOME/bin:$PATH
$ source .bash_profile
$ spark-shell
Spark context Web UI available at http://127.0.0.1:4040
Spark context available as 'sc' (master = local[*], app id = local-1511009262380).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.2.0
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_72)
Type in expressions to have them evaluated.
Type :help for more information.
scala>