Ambari搭建Hadoop集群

1.1 环境需求

1）3个节点的搭建

2）分别为master slave1 slave2

设置静态ip

[root@master ~]# cd /etc/sysconfig/network-scripts/

   [root@master network-scripts]# vi ifcfg-enp0s3

把动态#BOOTPROTO="dhcp"注释掉，在新增：

TYPE="Ethernet"

PROXY_METHOD="none"

BROWSER_ONLY="no"

#BOOTPROTO="dhcp"

DEFROUTE="yes"

IPV4_FAILURE_FATAL="no"

IPV6INIT="yes"

IPV6_AUTOCONF="yes"

IPV6_DEFROUTE="yes"

IPV6_FAILURE_FATAL="no"

IPV6_ADDR_GEN_MODE="stable-privacy"

NAME="enp0s3"

UUID="4e3c88bd-7ce2-4c45-9571-7d47abe12a06"

DEVICE="enp0s3"

ONBOOT="yes"

 

BOOTPROTO=static #静态IP

IPADDR=192.168.10.191 #本机地址

NETMASK=255.255.0.0 #子网掩码

GATEWAY=192.168.10.10 #默认网关

[root@master network-scripts]# service network restart

[root@master ~]# vi /etc/resolv.conf

nameserver 8.8.8.8

nameserver 8.8.4.4

[root@master network-scripts]# service network restart

链接通3个节点（每个节点都执行）

Jdk的安装：（官网下载jdk1.8上传到虚拟机/opt目录下并解压）

 

[root@opt ~]# tar xvf jdk-8u101-linux-x64.tar.gz

改名：

[root@opt ~]# mv jdk1.8.0_101 jdk-1.8.0

3）下载mysql的驱动包，上传到虚拟机的/opt目录下并解压

mysql-connector-java-5.1.46.tar.gz

[root@master opt]# tar xf mysql-connector-java-5.1.46.tar.gz

4）配置Java环境变量在最后添加

[root@master ~]# vi /etc/profile

#java1.8.0

export JAVA_HOME=/opt/jdk-1.8.0

export JRE_HOME=${JAVA_HOME}/jre

export CLASSPATH=.

export PATH=${JAVA_HOME}/bin:$PATH

5）使环境变量生效

[root@master ~]# source /etc/profile

6）发送到子节点

[root@master opt]# scp -r jdk-8u101-linux-x64.tar.gz

mysql-connector-java-5.1.46.tar.gz slave1:/opt

[root@master opt]# scp -r jdk-8u101-linux-x64.tar.gz

mysql-connector-java-5.1.46.tar.gz slave2:/opt

7）在子节点安装jdk及mysql驱动，步骤与master一样。

8）发送到子节点（环境变量）

[root@master ~]# scp -r /etc/profile slave1:/etc/profile

[root@master ~]# scp -r /etc/profile slave2:/etc/profile

使环境变量生效

[root@slave1 ~]# source /etc/profile

验证jdk的安装：

[root@master ~]# java -version

有自己的版本则安装成功。

搭建yum源与安装必备软件（主从节点）

选择阿里云的yum源及epel源，安装必备软件

yum -y install wget

cd /etc/yum.repos.d/

mkdir bak

mv C* bak

wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo

 

yum -y install vim make gcc gcc-c++  curl zip unzip ntp net-tools epel-release

（3）关闭防火墙与selinux机制、设置ssh优化(主从都执行)

关闭防火墙与selinux机制

systemctl stop firewalld.service

systemctl disable firewalld.service

setenforce 0

sed -i "s/enforcing/disabled/g" /etc/sysconfig/selinux

 

关闭SELINUX

修改配置文件关闭

vi /etc/selinux/config

SELINUX=disabled

免密登录的配置

秘密登录

在/root/下输入 ssh-keygen 会在/root/.ssh下生产两个文件

[root@master ~]# ssh-keygen

把那个.pub文件重命名为authorized_keys

[root@master ~]# cd /root/.ssh

[root@master .ssh]# mv id_rsa.pub authorized_keys

然后在子节点上创建/root/.ssh文件夹mkdir /root/.ssh

把上面的文件发送到子节点

时间同步的配置

ntp配置

   安装与启动 检查是否安装ntp： （所有节点安装配置）

[root@master ~]# rpm -q ntp

package ntp is not installed 

[root@master ~]# yum -y install ntp

Loaded plugins: fastestmirror ntp.x86_64 0:4.2.6p5-25.el7.centos  

[root@master ~]# systemctl enable ntpd 

Created symlink from /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service. 

[root@master ~]# systemctl start ntpd

2.配置

 配置前先使用命令：ntpdate -u cn.pool.ntp.org，同步服务器 （master主节点）

[root@master ~]# vim /etc/ntp.conf

#restrict 127.0.0.1

#restrict ::1

server 127.127.1.0 # local clock

fudge 127.127.1.0 stratum 10

# Hosts on local network are less restricted.

#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

restrict master mask 255.255.255.254 nomodify notrap

# Use public servers from the pool.ntp.org project.

# Please consider joining the pool (http://www.pool.ntp.org/join.html). 

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst 

[root@master ~]# service ntpd start  #启动ntpd时间服务器

Redirecting to /bin/systemctl start  ntpd.service

[root@master ~]# chkconfig ntpd on   #开机自启动

Note: Forwarding request to 'systemctl enable ntpd.service'.

[root@master ~]# ntpq -p

     remote     refid      st t when poll reach   delay   offset  jitter

==============================================================================

 ntp.wdc1.us.lea .STEP.     16 u - 1024   0    0.000    0.000   0.000

*ntp6.flashdance 192.36.143.130 2 u 57  64  377  353.810  -23.186  21.436

+ntp1.ams1.nl.le 130.133.1.10  2 u 40   64  377  160.311   -4.707  16.778

+ntp1.flashdance 194.58.202.20  2 u 60  64  377  351.746  -27.783  20.227

3.其他节点安装配置

  [root@slave ~]# vim /etc/ntp.conf

#restrict default nomodify notrap nopeer noquery

# Permit all access over the loopback interface.  This could

# be tightened as well, but to do so would effect some of

# the administrative functions.

#restrict 127.0.0.1

#restrict ::1

server master

# Hosts on local network are less restricted.

#restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap

# Use public servers from the pool.ntp.org project.

# Please consider joining the pool (http://www.pool.ntp.org/join.html).

#server 0.centos.pool.ntp.org iburst

#server 1.centos.pool.ntp.org iburst

#server 2.centos.pool.ntp.org iburst

#server 3.centos.pool.ntp.org iburst 

[root@master etc]# service ntpd start  #启动ntpd时间服务器

Redirecting to /bin/systemctl start  ntpd.service

[root@master etc]# chkconfig ntpd on   #开机自启动

Note: Forwarding request to 'systemctl enable ntpd.service'.

[root@slave1 ~]# ntpq -p

 

 remote       refid      st t when poll reach   delay   offset  jitter

==============================================================================

 ntp.wdc1.us.lea .STEP.     16 u    - 1024    0    0.000    0.000   0.000

*120.25.115.19   10.137.53.7 2 u 62  64  377   40.357   -0.273   1.102

+ntp1.ams1.nl.le 130.133.1.10 2 u 58   64  377  160.967    8.229   0.602

+ntp1.flashdance 194.58.202.20 2 u 19  64  377  351.830  -10.846   8.496

查看时间同步状态：

[root@master ~]#ntpstat

timeout

[root@slave1 ~]# ntpstat

unsynchronised

   polling server every 64 s

安装mysql（主节点master及子节点slave1安装）

安装mysql（ambari自带的数据库为PostgreSQL）下载mysql安装源，下载亚马逊的ambari源

 

wget http://dev.mysql.com/get/mysql-community-release-el7-5.noarch.rpm

rpm -ivh mysql-community-release-el7-5.noarch.rpm

yum -y install mysql-community-server

初始化数据库、创建数据库、设置ambari权限、复制java连接mysql的驱动（注意驱动的放置位置）

/usr/bin/mysql_install_db         //初始化数据库

如果有警告：把etc的my.cnf删了：

[root@master ~]# cd /etc

[root@master etc]# rm -rf my.cnf

 

在把usr下的my.cnf移动到/etc下：

[root@master etc]# cd /usr

[root@master usr]# ls

 

bin  etc  games  include  lib  lib64  libexec  local  my.cnf  sbin  share  src  tmp

[root@master usr]# mv my.cnf /etc

[root@master usr]# ls

bin  etc  games  include  lib  lib64  libexec  local  sbin  share  src  tmp

chown -R mysql:mysql /var/lib/mysql

systemctl start mysqld

systemctl enable mysqld

设置mysql的密码：

[root@master ~]# /usr/bin/mysqladmin -u root password 'admin'    

Warning: Using a password on the command line interface can be insecure.

[root@master ~]# /usr/bin/mysqladmin -u root -h master password 'admin'       

Warning: Using a password on the command line interface can be insecure.

 

登入数据库进行操作

mysql -uroot -p

mysql> create database ambari;

mysql> grant all privileges on ambari.* to 'ambari'@'master' identified by 'ambari';

mysql> flush privileges;

放置java连接mysql驱动：

mkdir /usr/share/java

cp /opt/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/share/java/

安装ambari

获取亚马逊云的repo源

cd /etc/yum.repos.d/

wget http://s3.amazonaws.com/public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.0/ambari.repo

 

yum -y clean all

yum makecache

用yum安装

yum -y install ambari-server

 

在mysql中的ambari库中操作：（主master操作）

 

[root@master ~]# mysql -uroot -p

use ambari;

source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;

导入驱动

cp /opt/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar /usr/lib/ambari-server/

配置ambari

ambari-server setup

括号里的为默认选项 

用户名和密码均为ambari，主机为master或主机的ip

设置成功

echo 'server.startup.web.timeout=120' >> /etc/ambari-server/conf/ambari.properties

离线安装agent（可选可不选）

[root@master yum.repos.d]# scp ambari.repo slave1:`pwd`

[root@master yum.repos.d]# scp ambari.repo slave2:`pwd`

（三个节点都安装）

yum install ambari-agent -y && systemctl enable ambari-agent

systemctl start ambari-agent && systemctl status ambari-agent

启动ambari

[root@master ~]# ambari-server start

页面的搭建

 

访问ambari

 访问ambari

输入主机ip地址、端口8080、用户名与密码均为admin

进行访问

点击Launch Install Wizard

创建一个name：我的是hadoop

默认以上

Remove掉多余的

写入主机名或ip地址

 把秘钥考到里面：[root@master ~]# cat /root/.ssh/id_rsa

如果失败继续reply

选择你要安装的服务

 

默认即可

路径有home的，把home路径都替换掉

点报警的选项，查看报错

hdfs和yarn是ambari的bug，无法读取home目录选择其他目录就行

密码都是hive

密码都是oozie

密码都是admin

警告是配置过低，可以强制操作

点击Deploy

导入java连接mysql驱动（这个要执行）

[root@master ~]# cd /var/lib/ambari-server

[root@master ambari-server]#

cp /opt/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar

[root@master ambari-server]# cd resources/

[root@master resources]# cp /opt/mysql-connector-java-5.1.46/mysql-connector-java-5.1.46.jar  /var/lib/ambari-server/resources/

[root@master resources]# mv mysql-connector-java-5.1.46.jar mysql-connector-java.jar

如果安装失败 检查mysql是否还存在：mysql -uroot -p 登录 ：若登录失败，则重新安装mysql：

卸载mysql

 rpm -e mysql-community-release-el7-5.noarch

yum -y remove mysql-community-server

安装mysql

  rpm -ivh mysql-community-release-el7-5.noarch.rpm

yum -y install mysql-community-server

/usr/bin/mysql_install_db

chown -R mysql:mysql /var/lib/mysql

systemctl start mysqld

systemctl enable mysqld

mysql -uroot -p         

继续Retry

有警告：手动启一下服务；

这是直接成功启动所有服务的安装

 

错误总结

1mysql报错

1.初始化mysql数据库时报警告

New default config file was created as /usr/my.cnf and

will be used by default by the server when you start it.

You may edit this file to change server settings

WARNING: Default config file /etc/my.cnf exists on the system

This file will be read by default by the MySQL server

If you do not want to use this, either remove it, or use the

--defaults-file argument to mysqld_safe when starting the server

解决办法：

把/etc下的my.cnf删了,然后把/usr下的my.cnf移动到/etc下

心跳丢失报错

1.无心跳的机器# python -V，发现python版本不适2.7.5，与其他机器稍有差别，修改方法：

此python版本默认可能使用ssl验证，

修改vim /etc/python/cert-verification.cfg
[https]
verify=disable

2.重启服务丢失心跳：

解决办法：（所有节点）

 [root@master ~]# vim /etc/ambari-agent/conf/ambari-agent.ini

加入

[security]

force_https_protocol=PROTOCOL_TLSv1_2

重启ambari-agent restart

 

时间同步报错

1.时间同步超时错误

[root@slave2 ~]# ntpq -p

localhost: timed out, nothing received

***Request timed out

解决办法：

[root@slave2 ~]# yum -y remove ntp

重新装