Kubernetes高可用集群二进制部署(Runtime Docker)
Kubernetes高可用集群二进制部署（Runtime Docker）
Kubernetes（简称为：k8s）是Google在2014年6月开源的一个容器集群管理系统，使用Go语言开发，用于管理云平台中多个主机上的容器化的应用，Kubernetes的目标是让部署容器化的应用简单并且高效,Kubernetes提供了资源调度、部署管理、服务发现、扩容缩容、监控，维护等一整套功能，努力成为跨主机集群的自动部署、扩展以及运行应用程序容器的平台。 它支持一系列容器工具, 包括Docker、Containerd等。
一、集群环境准备
1.1 主机规划
| 主机IP地址 | 主机名 | 主机配置 | 主机角色 | 软件列表 |
|---|---|---|---|---|
| 192.168.1.241 | k8s-master01 | 4C16G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce |
| 192.168.1.247 | k8s-master02 | 2C8G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce |
| 192.168.1.248 | k8s-master03 | 4C16G | master | kube-apiserver、kube-controller-manager、kube-scheduler、etcd、kubelet、kube-proxy、docker-ce |
| 192.168.10.240 | k8s-worker01 | 4C16G | worker | kubelet、kube-proxy、docker-ce |
| 192.168.10.220 | k8s-worker02 | 4C16G | worker | kubelet、kube-proxy、docker-ce |
| 192.168.1.241 | k8s-master01 | 4C16G | LB | haproxy、keepalived |
| 192.168.1.247 | k8s-master02 | 2C8G | LB | haproxy、keepalived |
| 192.168.1.100 | / | / | VIP(虚拟IP) |
1.2 软件版本
| 软件名称 | 版本 | 备注 |
|---|---|---|
| CentOS7 | kernel版本：5.16 | |
| kubernetes | v1.28 | |
| etcd | v3.5.2 | |
| calico | v3.19.4 | |
| coredns | v1.8.4 | |
| docker-ce | 18.09 | |
| haproxy | 5.18 | YUM源默认 |
| keepalived | 3.5 | YUM源默认 |
1.3 网络分配
| 网络名称 | 网段 | 备注 |
|---|---|---|
| Node网络 | 192.168.1.0/24 | |
| Service网络 | 10.96.0.0/16 | |
| Pod网络 | 10.244.0.0/16 |
二、集群部署
2.1主机准备
2.1.1 主机名设置
hostnamectl set-hostname xxx
TIPS:
主机名参见1.1小节主机规划表
2.1.2 主机与IP地址解析
cat >> /etc/hosts << EOF
192.168.1.241 k8s-master01
192.168.1.247 k8s-master02
192.168.1.248 k8s-master03
192.168.1.240 k8s-worker01
192.168.1.220 k8s-worker02
EOF
2.1.3 主机安全设置
2.1.3.1 关闭防火墙
systemctl stop firewalld
systemctl disable firewalld
firewall-cmd --state
2.1.3.2 关闭selinux
setenforce 0
sed -ri 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
sestatus
2.1.4 交换分区设置
swapoff -a
sed -ri 's/.*swap.*/#&/' /etc/fstab
echo "vm.swappiness=0" >> /etc/sysctl.conf
sysctl -p
2.1.5 主机系统时间同步
- 安装软件
yum -y install ntpdate
- 制定时间同步计划任务
crontab -e
0 */1 * * * ntpdate time1.aliyun.com
- 检查时间同步任务是否设置成功
crontab -l
2.1.6 主机系统优化
- limit优化
ulimit -SHn 65535
cat <<EOF >> /etc/security/limits.conf
soft nofile 655360
hard nofile 131072
soft nproc 655350
hard nproc 655350
soft memlock unlimited
hard memlock unlimited
EOF
2.1.7 ipvs管理工具安装及模块加载
- 为集群节点安装，负载均衡节点不用安装
yum install -y ipvsadm ipset sysstat conntrack libseccomp
- 创建 /etc/modules-load.d/ipvs.conf 并加入以下内容
cat >/etc/modules-load.d/ipvs.conf <<EOF
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
- 设置为开机启动
systemctl enable --now systemd-modules-load.service
2.1.8 Linux内核升级
- 在所有节点中安装，升级操作系统内核
yum -y install perl
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum -y install https://www.elrepo.org/elrepo-release-7.0-4.el7.elrepo.noarch.rpm
yum --enablerepo="elrepo-kernel" -y install kernel-ml.x86_64
grub2-set-default 0
grub2-mkconfig -o /boot/grub2/grub.cfg
2.1.9 Linux内核优化
cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 131072
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system
- 所有节点配置完内核后，重启服务器，保证重启后内核依旧加载
reboot -h now
- 重启后查看结果
lsmod | grep --color=auto -e ip_vs -e nf_conntrack
2.1.10 其它工具安装(选装)
yum install -y wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git lrzsz
2.2 负载均衡器准备
2.2.1 安装haproxy与keepalived
yum install -y haproxy keepalived
2.2.2 HAProxy配置
cat >/etc/haproxy/haproxy.cfg<<"EOF"
global
maxconn 10000
ulimit-n 16384
log 127.0.0.1 local0 err
stats timeout 30s
defaults
log global
mode http
option httplog
timeout connect 5000
timeout client 50000
timeout server 50000
timeout http-request 15s
timeout http-keep-alive 15s
frontend monitor-in
bind *:33305
mode http
option httplog
monitor-uri /monitor
frontend k8s-master
bind 0.0.0.0:6443
bind 127.0.0.1:6443
mode tcp
option tcplog
tcp-request inspect-delay 5s
default_backend k8s-master
backend k8s-master
mode tcp
option tcplog
option tcp-check
balance roundrobin
default-server inter 10s downinter 5s rise 10 fall 5 slowstart 60s maxconn 5000 maxqueue 10000 weight 100
server k8s-master01 192.168.1.241:16443 check
server k8s-master02 192.168.1.247:16443 check
server k8s-master03 192.168.1.248:16443 check
EOF
2.2.3 KeepAlived
主从配置不一致，需要注意。
- ha1
cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state MASTER
interface ens192
mcast_src_ip 192.168.1.241
virtual_router_id 51
priority 100
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.100
}
track_script {
chk_apiserver
}
}
EOF
- ha2
cat >/etc/keepalived/keepalived.conf<<"EOF"
! Configuration File for keepalived
global_defs {
router_id LVS_DEVEL
script_user root
enable_script_security
}
vrrp_script chk_apiserver {
script "/etc/keepalived/check_apiserver.sh"
interval 5
weight -5
fall 2
rise 1
}
vrrp_instance VI_1 {
state BACKUP
interface ens192
mcast_src_ip 192.168.1.247
virtual_router_id 51
priority 99
advert_int 2
authentication {
auth_type PASS
auth_pass K8SHA_KA_AUTH
}
virtual_ipaddress {
192.168.1.100
}
track_script {
chk_apiserver
}
}
EOF
2.2.4 健康检查脚本
ha1及ha2均要配置
cat > /etc/keepalived/check_apiserver.sh <<"EOF"
#!/bin/bash
err=0
for k in $(seq 1 3)
do
check_code=$(pgrep haproxy)
if [[ $check_code == "" ]]; then
err=$(expr $err + 1)
sleep 1
continue
else
err=0
break
fi
done
if [[ $err != "0" ]]; then
echo "systemctl stop keepalived"
/usr/bin/systemctl stop keepalived
exit 1
else
exit 0
fi
EOF
chmod +x /etc/keepalived/check_apiserver.sh
2.2.5 启动服务并验证
- 将haproxy,keepalived服务设置成开机自启动
systemctl daemon-reload
systemctl enable --now haproxy
systemctl enable --now keepalived
- 验证haproxy,keepalived是否配置成功
ip address show
TIPS:
- haproxy常用命令如下:
systemctl status haproxy
systemctl start haproxy
systemctl restart haproxy
systemctl reload haproxy
systemctl stop haproxy
- keepalived常用命令如下:
systemctl status keepalived
systemctl start keepalived
systemctl restart keepalived
systemctl reload keepalived
systemctl stop keepalived
2.3 配置免密登录
在k8s-master01,k8s-master02,k8s-master03,k8s-worker01,k8s-worker02节点上操作
ssh-keygen
ssh-copy-id root@k8s-master01
ssh-copy-id root@k8s-master02
ssh-copy-id root@k8s-master03
ssh-copy-id root@k8s-worker01
- 验证免密登录是否OK
ssh root@k8s-master01
ssh root@k8s-master02
ssh root@k8s-master03
ssh root@k8s-worker01
ssh root@k8s-worker02
2.4 部署ETCD集群
先在在k8s-master01节点上操作,然后分发至k8s-master02,k8s-master03
2.4.1 创建工作目录
mkdir -p /data/k8s-work
2.4.2 获取cfssl工具
cd /data/k8s-work
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64
#或者
wget https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl_linux-amd64
wget https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssljson_linux-amd64
wget https://github.com/cloudflare/cfssl/releases/download/1.2.0/cfssl-certinfo_linux-amd64
TIPS:
cfssl是使用go编写，由CloudFlare开源的一款PKI/TLS工具。主要程序有：
- cfssl，是CFSSL的命令行工具
- cfssljson用来从cfssl程序获取JSON输出，并将证书，密钥，CSR和bundle写入文件中。
- 授予cfssl可执行权限
chmod +x cfssl*
mv cfssl_linux-amd64 /usr/local/bin/cfssl
mv cfssljson_linux-amd64 /usr/local/bin/cfssljson
mv cfssl-certinfo_linux-amd64 /usr/local/bin/cfssl-certinfo
- 执行如下命令,检测cfssl是否安装成功
cfssl version
- 结果如下所示,即为安装成功
Version: 1.2.0
Revision: dev
Runtime: go1.6
2.4.3 创建CA证书
2.4.3.1 配置ca证书请求文件
cat > ca-csr.json <<"EOF"
{
"CN": "kubernetes",
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "WuHan",
"L": "WuHan",
"O": "kubeXiaoFanQie",
"OU": "CN"
}
],
"ca": {
"expiry": "87600h"
}
}
EOF
2.4.3.2 创建ca证书
cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2.4.3.3 配置ca证书策略
cat > ca-config.json <<"EOF"
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}
EOF
TIPS:
server auth 表示client可以对使用该ca对server提供的证书进行验证
client auth 表示server可以使用该ca对client提供的证书进行验证
2.4.4 创建etcd证书
2.4.4.1 配置etcd请求文件
cat > etcd-csr.json <<"EOF"
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.1.241",
"192.168.1.247",
"192.168.1.248"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [{
"C": "CN",
"ST": "WuHan",
"L": "WuHan",
"O": "kubeXiaoFanQ",
"OU": "CN"
}]
}
EOF
2.4.4.2 生成etcd证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd
- 查看证书文件是否生成
ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem etcd.csr etcd-csr.json etcd-key.pem etcd.pem
2.4.5 部署etcd集群
2.4.5.1 下载etcd软件包
wget https://github.com/etcd-io/etcd/releases/download/v3.5.2/etcd-v3.5.2-linux-amd64.tar.gz
2.4.5.2 安装etcd软件
tar -xvf etcd-v3.5.2-linux-amd64.tar.gz
cp -p etcd-v3.5.2-linux-amd64/etcd* /usr/local/bin/
2.4.5.3 分发etcd软件
scp etcd-v3.5.2-linux-amd64/etcd* k8s-master02:/usr/local/bin/
scp etcd-v3.5.2-linux-amd64/etcd* k8s-master03:/usr/local/bin/
2.4.5.4 创建配置文件
- 创建etcd配置文件目录
mkdir /etc/etcd
- 创建服务配置文件目录
mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd
- 移动证书至etcd证书目录
cd /data/k8s-work
cp ca*.pem /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl
- 执行如下命令生成etcd.conf配置信息
cat > /etc/etcd/etcd.conf <<"EOF"
#[Member]
ETCD_NAME="etcd1"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.241:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.241:2379,https://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.241:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.241:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.241:2380,etcd2=https://192.168.1.247:2380,etcd3=https://192.168.1.248:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_CLIENT_CERT_AUTH=true
ETCD_TRUSTED_CA_FILE=/etc/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/etcd/ssl/etcd.pem
ETCD_KEY_FILE=/etc/etcd/ssl/etcd-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=true
ETCD_PEER_CERT_FILE=/etc/etcd/ssl/etcd.pem
ETCD_PEER_KEY_FILE=/etc/etcd/ssl/etcd-key.pem
ETCD_PEER_TRUSTED_CA_FILE=/etc/etcd/ssl/ca.pem
EOF
配置项说明：
ETCD_NAME：节点名称，集群中唯一
ETCD_DATA_DIR：数据目录
ETCD_LISTEN_PEER_URLS：集群通信监听地址
ETCD_LISTEN_CLIENT_URLS：客户端访问监听地址
ETCD_INITIAL_ADVERTISE_PEER_URLS：集群通告地址
ETCD_ADVERTISE_CLIENT_URLS：客户端通告地址
ETCD_INITIAL_CLUSTER：集群节点地址
ETCD_INITIAL_CLUSTER_TOKEN：集群Token
ETCD_INITIAL_CLUSTER_STATE：加入集群的当前状态，new是新集群，existing表示加入已有集群
2.4.5.5 创建服务配置文件
- 执行如下命令生成etcd.service配置信息
cat > /etc/systemd/system/etcd.service <<"EOF"
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
WorkingDirectory=/var/lib/etcd/
ExecStart=/usr/local/bin/etcd
Restart=on-failure
RestartSec=5
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
2.4.5.6 同步etcd配置到集群其它master节点
- 创建目录
mkdir -p /etc/etcd
mkdir -p /etc/etcd/ssl
mkdir -p /var/lib/etcd/default.etcd
- 服务配置文件,需要修改etcd节点名称及IP地址
scp /etc/etcd/etcd.conf k8s-master02:/etc/etcd/
scp /etc/etcd/etcd.conf k8s-master03:/etc/etcd/
- 修改k8s-master02节点etcd.conf配置如下
cat /etc/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd2"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.247:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.247:2379,https://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.247:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.247:2379"
ETCD_INITIAL_CLUSTER="etcd1=https://192.168.1.241:2380,etcd2=https://192.168.1.247:2380,etcd3=https://192.168.1.248:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
- 修改k8s-master03节点etcd.conf配置如下
cat /etc/etcd/etcd.conf
#[Member]
ETCD_NAME="etcd3"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.1.248:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.1.248:2379,https://127.0.0.1:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.1.248:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.1.248:2379"
ETCD_INITIAL_CLUSTER="etcd1=