Ambari集群搭建

 

一.软件要求

1.yum, rpm

2.scp curl unzip tar wget

3.jdk1.8.0

---

二.前提环境准备

2.1 ssh免密

生成master机器的ssh密钥

# 生成master机器的公钥和密钥
[@hadoop.master01.com hadoop_install] ssh-keygen -t rsa

修改ssh登陆配置(禁用严格检查主机) 

[@hadoop.master01.com hadoop_install] vim ~/.ssh/config
StrictHostKeyChecking no
UserKnownHostsFile /dev/null

将master机器的公钥推送到其他机器上，${pass} ${ip}填写你的主机ip和主机密码，如果机器较多可以考虑写个脚本轮询即可

[@hadoop.master01.com hadoop_install] sshpass -p ${pass} ssh-copy-id root@${ip}

 

2.2修改域名解析文件以及自己的主机名

修改master机器上的/etc/hosts文件

[@hadoop.master01.com hadoop_install] vim /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.134.65.173 hadoop.master01.com
10.143.46.213 hadoop.master02.com
10.139.53.168 hadoop.slave01.com
10.139.50.172 hadoop.slave02.com
10.142.103.174 hadoop.slave03.com

利用scp命令推送到其他机器上,机器比较多的时候利用脚本轮询 

[@hadoop.master01.com hadoop_install] scp /etc/hosts ${ip}:/etc/hosts

执行hostname命令（临时修改）和/etc/hostname文件 （永久修改），集群里面的其他机器也需要修改，机器较多的时候写脚本执行

[@hadoop.master01.com hadoop_install] hostname hadoop.master01.com
[@hadoop.master01.com hadoop_install] vim /etc/hostname
hadoop.master01.com

 

---

三.制作本地YUM源

3.1 安装制作本地源工具

[@hadoop.master01.com hadoop_install] yum install yum-utils createrepo

3.2 安装HTTP服务器

[@hadoop.master01.com hadoop_install] yum install httpd -y

# 设置为开机自启动并且启动httpd服务
[@hadoop.master01.com hadoop_install] systemctl enable httpd && systemctl start httpd

3.3 下载系统对应的安装包

由于集群机器通常处于内网环境下，因此我们需要在外网机器上下载好对应的RPM包，最后通过lszrz命令上传。如果没有wget命令可以直接把URL复制到浏览器中，会自动执行下载

wget http://public-repo-1.hortonworks.com/ambari/centos7/2.x/updates/2.6.2.2/ambari-2.6.2.2-centos7.tar.gz
wget http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.6.4.0/HDP-2.6.4.0-centos7-rpm.tar.gz
wget http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.22/repos/centos7/HDP-UTILS-1.1.0.22-centos7.tar.gz

3.5 解压安装包

tar -zxvf ambari-2.6.2.2-centos7.tar.gz -C /var/www/html
tar -zxvf HDP-2.6.4.0-centos7-rpm.tar.gz -C /var/www/html/hdp/
tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz  -C /var/www/html/hdp/HDP-UTILS/

3.4 配置好对应的本地源（注意baseurl和gpgcheck里面写自己的repository的ip地址）

ambari.repo

#VERSION_NUMBER=2.6.2.2
[ambari-2.6.2.2]
name=ambari Version - ambari-2.6.2.2
baseurl=http://hadoop.master01.com/ambari/centos7/2.6.2.2-1
gpgcheck=1
gpgkey=http://hadoop.master01.com/ambari/centos7/2.6.2.2-1/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

 hdp.repo

#VERSION_NUMBER=2.6.4.0-91
[HDP-2.6.4.0]
name=HDP Version - HDP-2.6.4.0
baseurl=http://hadoop.master01.com/hdp/HDP/centos7/2.6.4.0-91
gpgcheck=1
gpgkey=http://hadoop.master01.com/hdp/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

[HDP-UTILS-1.1.0.22]
name=HDP-UTILS Version - HDP-UTILS-1.1.0.22
baseurl=http://hadoop.master01.com/hdp/HDP-UTILS
gpgcheck=1
gpgkey=http://hadoop.master01.com/hdp/HDP/centos7/2.6.4.0-91/RPM-GPG-KEY/RPM-GPG-KEY-Jenkins
enabled=1
priority=1

---

四.安装

4.1 Ambari-server(只在namenode节点上安装，standbynamenode上不用)

yum install ambari-server -y

4.2 安装java(直接在oracle java官网下载, 然后解压到/usr/local/下)

关于java这个安装操作请在集群上的所有机器上执行,如果机器数量较多的话可以借助scp功能推送

# 删除自带的java
[@hadoop.master01.com hadoop_install] yum remove java

# 删除自带的java开发包
[@hadoop.master01.com hadoop_install] yum remove java-devel

# 解压我们自己的jdk
[@hadoop.master01.com hadoop_install] tar xvf jdk-8u191-linux-x64.tar.gz -C /usr/local

4.3 开始配置Ambari-server

[@hadoop.master01.com jdk1.8.0_191]# ambari-server setup
Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
SELinux status is 'disabled'
Ambari-server daemon is configured to run under user 'ambari'. Change this setting [y/n] (n)? y
Enter user account for ambari-server daemon (ambari):
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
Do you want to change Oracle JDK [y/n] (n)? y
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK
==============================================================================
Enter choice (1): 3
WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts.
WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts.
Path to JAVA_HOME: /usr/local/jdk1.8.0_191       
Validating JDK on Ambari Server...done.
Checking GPL software agreement...
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
==============================================================================
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
==============================================================================
Enter choice (1): 1
Database admin user (postgres): 
Database name (ambari): 
Postgres schema (ambari): 
Username (ambari): 
Enter Database Password (123456): 
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Backup for pg_hba found, reconfiguration not required
Creating schema and user...
done.
Creating tables...
done.
Extracting system views...
............
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

4.4 启动ambari-server

请确保以下目录的所属用户是ambari，否则可能启动不了

/var/run/ambari-server/
/var/log/ambari-server/

启动ambari-server

ambari-server start

启动结束后可以在浏览器中输入10.134.65.173:8080,检查是否成功，这里的10.134.65.173是我自己部署ambari-server的机器ip，如果你需要检查的话替换以下ip便好.看到下面这个页面，就说明启动成功了

Username:admin

Password:admin

---

五.Hadoop服务安装

 

 5.1 填写集群的名称

5.2 选择Hadoop集成的版本，并填写repository地址

5.3 填写集群的主机域名和Master机器（部署ambari-server的那台）私钥

查看私钥

[@hadoop.master01.com jdk1.8.0_191] cat ~/.ssh/id_rsa

-----BEGIN RSA PRIVATE KEY-----
MIIEpgIBAAKCAQEA5ansWVx8CKLj5tJGdu5rhd9+RJzvoWJFwTQ2+1eX8P0hrvsN
3hVJHF67tFqK4K5RQCWa9nQqJPDQO8Of3uo9gncAcfF4DHEA/dubYQc52iXBurC0
ZPtvxCH5L1lXt6FYQiIYphjpi3raighYCQUd+tfMF1yC8MXxkrHO+pfStLG2Xf4X
UgMfd1aGZjDrNNg+wL552AY3AzpzaoHTaTW08emqumgFwky3pMKV4xsM+8Biwme9
TL8cI6TlmXQXSixqGz2mQYMcK68czcNUe5WR0tp90oHt2sDa4QLoQmDvpqk9U/+D
Qp2QEjERUr4CLiUDKyhsEXZKyfn2A8X47E1dUwIDAQABAoIBAQDVnShdsb0YV1H9
FqY95uMRwN/0N7hXdk2FN+jP/RgxKXDUDWVHpKq8/Yw/Y1v7QYgHTzMEeuXcODSm
8fzYXS1bFQ7P7JYJlG4ksMekBxGTSz0XRVZ0+r1pQ2Q/2yzQevzZfldI3rETb/Cn
JL2lniHVeDS3oe+UvbhFBqeKw/5S25ldC9ROGw0OZgl52w1d+J/q38O+VhiZdnj6
Sb+vZX0QwG1zu5aV9k94iGl9DhyJRPKjOCmPc/mgLP6k8/mYTFIB8ntJH1LcQvF9
81ovp/zeQt4FwZf8cGnnwuwaWiTFiEd4iB8xQl0hBTpgW/ijmlJ6qwe1SOOA0zwp
qm/RWzuZAoGBAPzBOCNHhJiydcyuiLhBiG8dTuDd3GQTUUy31qZ1qeHaFSvir1fF
UZoowuPV+fgN7PFyzl1tSjOwxrxiizZF5iARFZ3Sw3TuxnUihjjytxvjn6VxwPKI
9ICf3q2xlYKuezWtF676ufnPRDaXgQFoRVjlVq5nkwaohzdp7k0/pGNHAoGBAOic
zljTQi9KwC2vBS/8QL9O1sBzGAjN2vAF9ThlForWKFZKNF4JW5nd8+LLhxXY3hSv
ria4lQkFUy1vdcJaimPVPynvZRqgPH6GaqK/YDyu8kpTRkOO4HJTiFuv9Z16JasW
a+Ot8F6BPRRLQas7PgtSvmh9Egjg6Jq9gxUxPEOVAoGBAKEgwBvldMcvRV+FiE0G
IebL5UP+y4fUtMhWjfUyH2mGcGa4rRUkCiBfidZay27x2bIWgS4YBOys7hScMakY
qFqa6Y/iE45QxZ1Ys/5L5KVX16dNvDYhTtVMlVVfVzOm7Y7LWMTIu5RgKFkw/iGa
ilLgaQDnsFfXSdPuqOcAAljzAoGBAOMW+shy7oFW+8ldvbW8EcXzBOr7MfKebei4
GE/HOLuwGc8sQHWxsxDDQ+8c/bNRdnZBkMZSm4UkArEvyIpGdCbEsBaTlWPqHUT0
mzRpm12C+h2fsNARs4NJk6zXWvUzo4Skrg2n6SuKBoRCeEjf3FGXpTZDLBEo6Uzw
SQGugIQFAoGBAPdIJIAzdJ9TyKKvQCOxqXoo5iYrQoqM6hy8MXYJeGWLbU6aszWb
yUO5smURZgFfm0LddP56Ubx7SQ5aBBVLgXaGTeYNFKuWWXqCim4hSx0+/r6W+6KJ
HozWYErDghVLrofED/GLHuQCFbLvkXQOvvKbtrGlfoX6Rn5/WK00SqjL
-----END RSA PRIVATE KEY-----

5.4 确认主机，检查主机（这里大家可能会失败），

需要在/etc/ambari-agent/conf/ambari-agent.ini中增加一行配置（在集群中的所有机器中操作）

[security]
force_https_protocol=PROTOCOL_TLSv1_2

增加完了别忘了重启ambari-agent(注意不是ambari-server,在集群中的所有机器中操作)

ambari-agent restart

看到下图的时候就代表检查主机成功！！！

 

后面的内容是关于hadoop集群比较简单的角色划分部分了。。自己可以尝试下，未完待续