基于Contos7.X+CDH6.2.0大数据平台搭建(上)
目录
一、CDH6.2.0官网介绍
二、CDH6.2.0安装前环境的部署
三、CDH6.2.0安装
一、CDH6.2官网介绍：
CDH(Cloudera's Distribution,including Apache Hadoop).是Haoop众多分支中的一种大数据，简称CDH，基于最稳定版本的Apache Hadoop，有许多补丁、向后移植和更新，Cloudera公司以多种不同的形式进行发布。包括源码和二进制tar文件，、RPM等等。CDH是Apche2.0许可下发布的自由软件，用户可以直接从官方https://www.cloudera.com/downloads/cdh/6-2-0.html上获取。
二、CDH6.2.0安装前环境的部署
CDH的部署方式：联网安装，离线包安装。
1、CM6.2.0以及CDH6.2.0的安装：
系统环境：
- 主机规划
| 主机名 | 角色 | IP地址 | 用途 | 配置 |
|---|---|---|---|---|
| cdh01 | 主从节点 | 192.168.80.201 | CM、Cloudera Server、mysql、Agent | 4C/16G/150G |
| cdh02 | 从节点 | 192.168.80.202 | Cloudera Agen | 4C/16G/150G |
| cdh03 | 从节点 | 192.168.80.203 | Cloudera Agen | 4C/16G/150G |
注意：主机的内存最好在10G以上，才能保证环境的正常运行。
2、提前下载相关的RPM安装包以及封装的组件的Parcel包。
Cloudera Manager以及相关依赖环境、jdk、agent以及server的下载地址：
- cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
- cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
- cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
- cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
- oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
3、在主节点上配置Cloudera Manager仓库（添加存储库的时候会用到）。
[root@cdh01 ~]# yum -y install httpd createrepo
[root@cdh01 ~]# systemctl start httpd && systemctl enable httpd
[root@cdh01 ~]# mkdir /opt/cloudera-repos/ -p
[root@cdh01 ~]# cd /opt/cloudera-repos/ && createrepo . #生成RPM元数据：
[root@cdh01 ~]# ll /opt/cloudera-repos/
cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
repodata
4、Centos7.2版本下的CDH6.2.0二进制Parcel包下载地址（下载与系统匹配的包）：https://archive.cloudera.com/cdh6/6.2.0/parcels/
- CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
- CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256
- manifest.json
主节点上创建/opt/cloudera/parcel-repo/ 导入parcel包、生成sha1mun值
[root@cdh01 parcel-repo]# ll
CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel
CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256
manifest.json
[root@cdh01 parcel-repo]# mv CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha256 CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha
[root@cdh01 parcel-repo]# sha1sum CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel | awk '{ print $1 }' > CDH-6.2.0-1.cdh6.2.0.p0.967373-el7.parcel.sha5、服务器进行系统环境设置：
每台服务器分别定义各自的hostname：cdh01、cdh02、cdh03
服务器分别把主机名与ip对应关系写进/etc/hosts上。（发送到所有节点）
# cat /etc/hosts
192.168.80.201 cdh01
192.168.80.202 cdh02
192.168.80.203 cdh03所有节点服务器永久关闭防火墙、selinux （阿里云虚拟机忽略此步骤）
systemctl stop firewalld.service
systemctl disable firewalld.service
setenforce 0
sed -i "s/^SELINUX=enforcing/SELINUX=disabled/g" /etc/selinux/config
所有服务器配置ntp服务，添加时间服务器，并启动时间服务器，开机启动，写入硬件时钟 (阿里云虚拟机忽略此步骤）
[root@cdh01 ~]# yum -y install ntp
[root@cdh01 ~]# sed -i 's/server 0.centos.pool.ntp.org iburst/#server 0.centos.pool.ntp.org iburst/g' #/etc/ntp.conf
[root@cdh01 ~]# sed -i 's/server 1.centos.pool.ntp.org iburst/#server 1.centos.pool.ntp.org iburst/g' #/etc/ntp.conf
[root@cdh01 ~]# sed -i 's/server 2.centos.pool.ntp.org iburst/#server 2.centos.pool.ntp.org iburst/g' #/etc/ntp.conf
[root@cdh01 ~]# sed -i 's/server 3.centos.pool.ntp.org iburst/#server 3.centos.pool.ntp.org iburst/g' #/etc/ntp.conf
[root@cdh01 ~]# sed -ie 's/#server 3.centos.pool.ntp.org iburst/#server 3.centos.pool.ntp.org iburst \nserver 10.17.87.8/g' /etc/ntp.conf
[root@cdh01 ~]# systemctl restart ntpd && systemctl enable ntpd && ntpq -p && hwclock -r
安装python2.7（centos7.2默认安装python2.7）(阿里云虚拟机忽略此步骤）
# python --version
Python 2.7.5
设置主节点服务器免密码登陆从节点服务器
[root@cdh01 ~]# ssh-keygen -t rsa
[root@cdh01 ~]# chmod 600 ~/.ssh/authorized_keys
[root@cdh01 ~]# ssh-copy-id root@cdh01
[root@cdh01 ~]# ssh-copy-id root@cdh02
[root@cdh01 ~]# ssh-copy-id root@cdh03
三、CDH6.2.0的安装：
1、主节点安装repo，并导入GPG key
[root@cdh03 ~]# wget https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/cloudera-manager.repo -P /etc/yum.repos.d/
[root@cdh03 ~]# rpm --import https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPM-GPG-KEY-cloudera
2、主节点上按照如下顺序进行JDK、CM、server、agent安装
[root@cdh01 cloudera-repos]# yum -y install oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
[root@cdh01 cloudera-repos]# vi /etc/profile
JAVA_HOM=E/root/jdk1.8.0
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PAT
[root@cdh01 cloudera-repos]# yum -y install cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
[root@cdh01 cloudera-repos]# yum -y install cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
[root@cdh01 cloudera-repos]# yum -y install cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
[root@cdh01 cloudera-repos]# yum -y install cloudera-manager-server-db-2-6.2.0-968826.el7.x86_64.rpm
3、从节点上按照如下顺序进行JDK、agent安装
cdh02从节点
[root@cdh02 cloudera-repos]# yum -y install oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
[root@cdh02 cloudera-repos]# vi /etc/profile
JAVA_HOM=E/root/jdk1.8.0
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
[root@cdh02 cloudera-repos]# yum -y install cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
[root@cdh02 cloudera-repos]# yum -y install cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
cdh03从节点
[root@cdh03 cloudera-repos]# yum -y install oracle-j2sdk1.8-1.8.0+update141-1.x86_64.rpm
[root@cdh03 cloudera-repos]# vi /etc/profile
JAVA_HOM=E/root/jdk1.8.0
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
[root@cdh03 cloudera-repos]# yum -y install cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
[root@cdh03 cloudera-repos]# yum -y install cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm4、修改agent的服务地址(所有节点)
打开并编辑 /etc/cloudera-scm-agent/config.ini 文件：
[root@cdh01]# vi /etc/cloudera-scm-agent/config.ini
...
[General]
# Hostname of the CM server.
#将 server_host=localhost 改为 master 节点主机名 cdh1-master
server_host=cdh01
# Port that the CM server is listening on.
server_port=7182
...5、主数据库5.6安装：
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
rpm -ivh mysql-community-release-el7-5.noarch.rpm
yum update
yum install mysql-server
systemctl start mysqld
systemctl enable mysqld
6、安装MySQL JDBC Driver
[root@cdh01 ~]# mkdir -p /usr/share/java/
[root@cdh01 ~]# cd mysql-connector-java-5.1.46
[root@cdh01 ~]# cp mysql-connector-java.jar /usr/share/java/7、给mysql数据库创建需要用到的库：scm、amon、rman、hue、metastore、sentry、nav、navms、oozie、hive
注意：如涉及Windows下链接Linux,注意网关ip授权问题
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY ' pan@123';
CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON metastore.* TO 'metastore'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'pan@123';
CREATE DATABASE hive DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON hive.* TO 'hive'@'%' IDENTIFIED BY 'pan@123';
flush privileges;
8、建立CM的数据库(建议数据库和CM在一台服务器上)：
若已经存在这使用命令删除 rm /etc/cloudera-scm-server/db.properties
[root@cdh01 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
Enter SCM password: #输入SCM的密码并且回车
写入如下：
JAVA_HOME=/usr/java/jdk1.8.0_141-cloudera
Verifying that we can write to /etc/cloudera-scm-server
Creating SCM configuration file in /etc/cloudera-scm-server
Executing: /usr/java/jdk1.8.0_141-cloudera/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.
[main] DbCommandExecutor INFO Successfully connected to database.
All done, your SCM database is configured correctly!
···