一、环境准备
1、集群规划
| hadoop01/192.168.1.151 | hadoop02/192.168.1.152 | hadoop03/192.168.1.153 | hadoop04/192.168.1.154 | hadoop05/192.168.1.155 | |
|---|---|---|---|---|---|
| ambari server | 是 | 否 | 否 | 否 | 否 |
| ambari agent | 是 | 是 | 是 | 是 | 是 |
| namenode | 是 | 是 | 否 | 否 | 否 |
| datanode | 否 | 否 | 是 | 是 | 是 |
| resourcemanager | 是 | 是 | 否 | 否 | 否 |
| journalnode | 是 | 是 | 是 | 是 | 是 |
| zookeeper | 是 | 是 | 是 | 否 | 否 |
01和02主要做管理机，03-05做数据数据节点，01-03做zookeeper集群。
2、安装一台lnmp备用
选择hadoop01
https://lnmp.org/install.html
wget -c http://soft.vpser.net/lnmp/lnmp1.5.tar.gz && tar zxf lnmp1.5.tar.gz && cd lnmp1.5 && ./install.sh lnmp
选择安装mysql5.7版本用于支持ambari
3、下载离线版所需文件
https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.2.0/bk_ambari-installation/content/ch_obtaining-public-repos.html
下载备用。
二、安装配置ambari
1、配置离线仓库地址
mkdir /home/wwwroot/default/ambari
mkdir /home/wwwroot/default/ambari/HDP-UTILS
cd /home/wwwroot/default/ambari
复制下载文件到相关目录并解压
tar -zxvf ambari-2.6.2.0-ubuntu16.tar.gz
tar -zxvf HDP-GPL-2.6.5.0-ubuntu16-gpl.tar.gz
tar -zxvf HDP-2.6.5.0-ubuntu16-deb.tar.gz
tar -zxvf HDP-UTILS-1.1.0.22-ubuntu16.tar.gz -C HDP-UTILS
开启nginx对ambari目录的浏览功能。
2、制作list文件并复制到各台agent主机
cd /etc/apt/sources.list.d
ambari.list
deb http://192.168.1.151/ambari/ambari/ubuntu16/2.6.2.0-155/ Ambari main
ambari-hdp.list
deb http://192.168.1.151/ambari/HDP/ubuntu16/2.6.5.0-292/ HDP main
deb http://192.168.1.151/ambari/HDP-GPL/ubuntu16/2.6.5.0-292/ HDP-GPL main
deb http://192.168.1.151/ambari/HDP-UTILS/HDP-UTILS/ubuntu16/1.1.0.22/ HDP-UTILS main
更新list
sudo apt-key adv --recv-keys --keyserver keyserver.ubuntu.com B9733A7A07513CAD
sudo apt-get update
apt-cache showpkg ambari-server
apt-cache showpkg ambari-agent
apt-cache showpkg ambari-metrics-assembly
3、安装ambari
sudo apt-get install ambari-server
root@hadoop01:/home# sudo apt-get install ambari-server
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libsensors4 postgresql postgresql-9.5 postgresql-client-9.5 postgresql-client-common postgresql-common postgresql-contrib-9.5 ssl-cert sysstat
Suggested packages:
lm-sensors postgresql-doc locales-all postgresql-doc-9.5 libdbd-pg-perl openssl-blacklist isag
The following NEW packages will be installed:
ambari-server libsensors4 postgresql postgresql-9.5 postgresql-client-9.5 postgresql-client-common postgresql-common postgresql-contrib-9.5 ssl-cert sysstat
0 upgraded, 10 newly installed, 0 to remove and 48 not upgraded.
Need to get 757 MB of archives.
After this operation, 831 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://192.168.1.151/ambari/ambari/ubuntu16/2.6.2.0-155 Ambari/main amd64 ambari-server amd64 2.6.2.0-155 [752 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 libsensors4 amd64 1:3.4.0-2 [28.4 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-common all 173ubuntu0.1 [28.4 kB]
Get:4 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-client-9.5 amd64 9.5.13-0ubuntu0.16.04 [871 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 ssl-cert all 1.0.37 [16.9 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-common all 173ubuntu0.1 [154 kB]
Get:7 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-9.5 amd64 9.5.13-0ubuntu0.16.04 [3,001 kB]
Get:8 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql all 9.5+173ubuntu0.1 [5,450 B]
Get:9 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 postgresql-contrib-9.5 amd64 9.5.13-0ubuntu0.16.04 [448 kB]
Get:10 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 sysstat amd64 11.2.0-1ubuntu0.2 [243 kB]
Fetched 757 MB in 12s (59.8 MB/s)
Preconfiguring packages ...
Selecting previously unselected package libsensors4:amd64.
(Reading database ... 70302 files and directories currently installed.)
Preparing to unpack .../libsensors4_1%3a3.4.0-2_amd64.deb ...
Unpacking libsensors4:amd64 (1:3.4.0-2) ...
Selecting previously unselected package postgresql-client-common.
Preparing to unpack .../postgresql-client-common_173ubuntu0.1_all.deb ...
Unpacking postgresql-client-common (173ubuntu0.1) ...
Selecting previously unselected package postgresql-client-9.5.
Preparing to unpack .../postgresql-client-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-client-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package ssl-cert.
Preparing to unpack .../ssl-cert_1.0.37_all.deb ...
Unpacking ssl-cert (1.0.37) ...
Selecting previously unselected package postgresql-common.
Preparing to unpack .../postgresql-common_173ubuntu0.1_all.deb ...
Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'
Unpacking postgresql-common (173ubuntu0.1) ...
Selecting previously unselected package postgresql-9.5.
Preparing to unpack .../postgresql-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package postgresql.
Preparing to unpack .../postgresql_9.5+173ubuntu0.1_all.deb ...
Unpacking postgresql (9.5+173ubuntu0.1) ...
Selecting previously unselected package postgresql-contrib-9.5.
Preparing to unpack .../postgresql-contrib-9.5_9.5.13-0ubuntu0.16.04_amd64.deb ...
Unpacking postgresql-contrib-9.5 (9.5.13-0ubuntu0.16.04) ...
Selecting previously unselected package sysstat.
Preparing to unpack .../sysstat_11.2.0-1ubuntu0.2_amd64.deb ...
Unpacking sysstat (11.2.0-1ubuntu0.2) ...
Selecting previously unselected package ambari-server.
Preparing to unpack .../ambari-server_2.6.2.0-155_amd64.deb ...
Unpacking ambari-server (2.6.2.0-155) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up libsensors4:amd64 (1:3.4.0-2) ...
Setting up postgresql-client-common (173ubuntu0.1) ...
Setting up postgresql-client-9.5 (9.5.13-0ubuntu0.16.04) ...
update-alternatives: using /usr/share/postgresql/9.5/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode
Setting up ssl-cert (1.0.37) ...
Setting up postgresql-common (173ubuntu0.1) ...
Adding user postgres to group ssl-cert
Creating config file /etc/postgresql-common/createcluster.conf with new version
Creating config file /etc/logrotate.d/postgresql-common with new version
Building PostgreSQL dictionaries from installed myspell/hunspell packages...
Removing obsolete dictionary files:
Setting up postgresql-9.5 (9.5.13-0ubuntu0.16.04) ...
Unescaped left brace in regex is deprecated, passed through in regex; marked by <-- HERE in m/(?5、初始配置
sudo ambari-server setup
使用自定义jdk
采用mysql数据库（生产环境采用内置postgresql不合适）
需要相关jdbc。
https://dev.mysql.com/downloads/connector/j/
在主节点把mysql数据库连接包拷贝在/var/lib/ambary-server/resources目录下并改名为mysql-jdbc-driver.jar
vim /etc/ambari-server/conf/ambari.properties
Configuring remote database connection properties...
WARNING: Before starting Ambari Server, you must run the following DDL against the database to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql
Proceed with configuring remote database connection properties [y/n] (y)? y
Extracting system views...
......ambari-admin-2.6.2.0.155.jar
.....
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.
三、配置ambari
1、启动ambari
root@hadoop01:~# sudo ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.........Unable to determine server PID. Retrying...
......Unable to determine server PID. Retrying...
......Unable to determine server PID. Retrying...
ERROR: Exiting with exit code -1.
REASON: Ambari Server java process died with exitcode 1. Check /var/log/ambari-server/ambari-server.out for more information.
出现错误。
导入mysql脚本
mysql -uroot -p
mysql> use ambari;
mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql;
再次启动
root@hadoop01:~# sudo ambari-server start
Using python /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/ambari-server.pid
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start.................................
Server started listening on 8080
DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.
采用admin/admin登录
四、配置hadoop
选择本地repository，填入本地url；
本地hdp-utils缺失文件错误（2.6文件几十k，2.5版本文件32mb，估计官网问题）。更换成公网地址。
或者用2.5的文件替换2.6的文件dist目录。
cat ~/.ssh/id_rsa
获取ssh密钥。
解决问题
etc/ambari-agent/conf/ambari-agent.ini
[security]
force_https_protocol=PROTOCOL_TLSv1_2
关闭thp
vi /etc/init.d/disable-transparent-hugepages
#!/bin/sh
### BEGIN INIT INFO
# Provides: disable-transparent-hugepages
# Required-Start:$local_fs
# Required-Stop:
# X-Start-Before:mongod mongodb-mms-automation-agent
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description: Disable Linux transparent huge pages, to improve
#database performance.
### END INIT INFO
case $1 in
start)
if [ -d /sys/kernel/mm/transparent_hugepage ]; then
thp_path=/sys/kernel/mm/transparent_hugepage
elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
thp_path=/sys/kernel/mm/redhat_transparent_hugepage
else
return 0
fi
echo 'never' > ${thp_path}/enabled
echo 'never' > ${thp_path}/defrag
unset thp_path
;;
esac
sudo chmod 755 /etc/init.d/disable-transparent-hugepages
sudo update-rc.d disable-transparent-hugepages defaults
选择需要的组件
hbase不是最新2.1.0版本。
经过各种坑，顺利完成。
配置高可用
高可用配置完成。