CentOS 7 安装 Spark 3.0 集群

参考文档：
https://www.cnblogs.com/aiaitie/p/9306604.html
https://www.cnblogs.com/cac2020/p/10644313.html

---

补充：

1、spark-env.sh的配置

export SCALA_HOME=/usr/local/scala/scala-2.13.1
export JAVA_HOME=/usr/local/java/jdk1.8
export HADOOP_HOME=/usr/local/hadoop/hadoop-3.2.0
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export SPARK_HOME=/usr/local/spark/spark-3.0.0-preview2-bin-hadoop3.2
export SPARK_MASTER_HOST=node100
export SPARK_MASTER_PORT=7077
export SPARK_WORKER_PORT=7078
export SPARK_WORKER_CORES=2
export SPARK_MASTER_WEBUI_PORT=8080
export SPARK_EXECUTOR_MEMORY=1G

slaves配置信息：

node200
node300

注意：
1、node100是主节点主机名，noden200和node300是从节点主机名
2、三个节点的配置信息都是一样的
3、SPARK_MASTER_IP被取消了，替代为SPARK_MASTER_HOST
4、SPARK_WORKER_PORT没有配置会报错
5、一开始配置SPARK_LOCAL_IP为主节点的真实IP，但是报错，删了就好了

---

一些报错：

1、SPARK_WORKER_PORT没有配置导致报错，加上即可，不要与SPARK_MASTER_PORT相同

WARN util.Utils: Service 'sparkWorker' could not bind on a random free port. You may check whether configuring an appropriate binding address.
ERROR util.SparkUncaughtExceptionHandler: Uncaught exception in thread Thread[main,5,main]
java.net.BindException: Cannot assign requested address: Service 'sparkWorker' failed after 16 retries (on a random free port)! Consider explicitly setting the appropriate binding address for the service 'sparkWorker' (for example spark.driver.bindAddress for SparkDriver) to the correct binding address.
	at sun.nio.ch.Net.bind0(Native Method)
	at sun.nio.ch.Net.bind(Net.java:433)
	at sun.nio.ch.Net.bind(Net.java:425)
	......

2、配置SPARK_LOCAL_IP为主节点的真实IP导致报错，我采取的方式比较简单粗暴，直接删了，hhhhhh

WARN Utils: Service 'sparkMaster' could not bind on port 7078. Attempting port 7079.
WARN Utils: Service 'sparkMaster' could not bind on port 7079. Attempting port 7080.
WARN Utils: Service 'sparkMaster' could not bind on port 7080. Attempting port 7081.
......

---

成功的界面：访问你的主节点IP:8080