二进制部署Kubernetes1.32.4最新版本高可用集群及附加组件
一、前言
在云原生技术席卷全球的今天，Kubernetes（K8s）已成为容器编排领域的事实标准。当大家都习惯了kubeadm、kubeasz等自动化工具一键部署的便利时，选择通过二进制方式手动搭建K8s集群更像是一场"知其然亦知其所以然"的深度修行。这种方式将带您穿透抽象层，直面etcd的分布式存储机制、kube-apiserver的RESTful接口设计、kubelet与CRI的交互细节，以及各个核心组件间的TLS双向认证体系。 本文将还原最原始的K8s部署逻辑，从零开始构建一个符合生产环境要求的高可用集群。不同于封装工具对底层实现的隐藏，二进制部署要求我们亲手配置每个组件的systemd服务单元，精确编排CA证书链的签发流程，甚至需要理解kube-proxy在不同模式下的iptables/ipvs规则生成机制。这种看似繁琐的过程，恰恰是理解Kubernetes调度器如何通过watch机制协调集群状态、控制器如何通过reconcile循环维持系统稳定的最佳实践路径。 通过本系列教程，您不仅将获得一个可横向扩展的集群架构（包括多master节点负载均衡与etcd集群部署方案），更重要的是建立起对K8s生态的立体认知。当面对证书过期、组件通信异常等生产环境常见故障时，这份通过二进制部署积累的排障经验，将成为您运维分布式系统的宝贵资产。让我们开始这场从源码到服务的探索之旅，揭开Kubernetes神秘面纱背后的工程之美。
就在北京时间2025年4月23日上午，K8S发布了最新版本K8S1.32.4版本，于是我就使用二进制的方式部署出了一套完整的集群，并且整理了最详细的笔记。
二、架构演示
三、基础环境
# 由于小编的学习环境实在是资源有限，所以我把Etcd集群，master集群和worker集群，负载均衡，高可用全部都放在了三个节点上面，如果环境资源允许的话大家是可以解耦的，因为这些服务不存在端口冲突的情况，于是我就放在了一起部署使用。
| 操作系统 | 主机名 | IP地址 | 系统配置 |
|---|---|---|---|
| Ubuntu22.04-LTS | node-exporter41 | 10.0.0.41/24 | 4Core/8G/100GB |
| Ubuntu22.04-LTS | node-exporter42 | 10.0.0.42/24 | 4Core/8G/100GB |
| Ubuntu22.04-LTS | node-exporter43 | 10.0.0.43/24 | 4Core/8G/100GB |
四、开始部署！！！
1.部署 Etcd 高可用集群
准备 Etcd 程序包
-
下载 Etcd 软件
wget https://github.com/etcd-io/etcd/releases/download/v3.5.21/etcd-v3.5.21-linux-amd64.tar.gz -
解压 Etcd 的二进制程序包到 PATH 环境变量路
tar -xf etcd-v3.5.21-linux-amd64.tar.gz -C /usr/local/bin etcd-v3.5.21-linux-amd64/etcd{,ctl} --strip-components=1验证
etcdctl版本etcdctl version -
将软件包下发到所有节点 使用
scp命令将/usr/local/bin目录下的etcd和etcdctl文件复制到其他节点。
准备 Etcd 的证书文件
-
安装 cfssl 证书管理工具
wget http://192.168.16.253/Resources/Prometheus/softwares/Etcd/oldboyedu-cfssl-v1.6.5.zip unzip oldboyedu-cfssl-v1.6.5.zip apt install rename rename -v "s/_1.6.5_linux_amd64//g" cfssl* chmod +x /usr/local/bin/cfssl* -
创建证书存储目录
mkdir -pv /caofacan/certs/etcd && cd /caofacan/certs/etcd/ -
生成证书的 CSR 文件（证书签发请求文件）
cat > etcd-ca-csr.json < -
生成 Etcd CA 证书和 CA 证书的 key
cfssl gencert -initca etcd-ca-csr.json | cfssljson -bare /caofacan/certs/etcd/etcd-ca -
生成 Etcd 证书的有效期配置文件
cat > ca-config.json < -
生成 apiserver 证书的 CSR 文件
cat > apiserver-csr.json < -
基于自建 ca 证书生成 apiServer 的证书文件
cfssl gencert \ -ca=/caofacan/certs/etcd/etcd-ca.pem \ -ca-key=/caofacan/certs/etcd/etcd-ca-key.pem \ -config=ca-config.json \ --hostname=10.200.0.1,10.0.0.240,kubernetes,kubernetes.default,kubernetes.default.svc,caofacan.com,kubernetes.default.svc.caofacan.com,10.0.0.41,10.0.0.42,10.0.0.43 \ --profile=kubernetes \ apiserver-csr.json | cfssljson -bare /caofacan/certs/etcd/etcd-server
创建 Etcd 集群各节点配置
-
node-exporter41 节点的配置文件
mkdir -pv /caofacan/softwares/etcd cat > /caofacan/softwares/etcd/etcd.config.yml << 'EOF' name: 'node-exporter41' data-dir: /var/lib/etcd listen-peer-urls: 'https://10.0.0.41:2380' listen-client-urls: 'https://10.0.0.41:2379,http://127.0.0.1:2379' initial-cluster: 'node-exporter41=https://10.0.0.41:2380,node-exporter42=https://10.0.0.42:2380,node-exporter43=https://10.0.0.43:2380' client-transport-security: cert-file: /caofacan/certs/etcd/etcd-server.pem key-file: /caofacan/certs/etcd/etcd-server-key.pem ca-file: /caofacan/certs/etcd/etcd-ca.pem client-cert-auth: true peer-transport-security: cert-file: /caofacan/certs/etcd/etcd-server.pem key-file: /caofacan/certs/etcd/etcd-server-key.pem ca-file: /caofacan/certs/etcd/etcd-ca.pem peer-client-cert-auth: true EOF -
node-exporter42 和 node-exporter43 节点的配置文件：类似 node-exporter41，需修改对应 IP 和节点名称。
配置 Etcd 启动脚本
在每个节点上创建 /usr/lib/systemd/system/etcd.service 文件：
[Unit]
Description=Jason Yin's Etcd Service
Documentation=https://coreos.com/etcd/docs/latest/
After=network.target
[Service]
ExecStart=/usr/local/bin/etcd --config-file=/caofacan/softwares/etcd/etcd.config.yml
Restart=on-failure
RestartSec=10
LimitNOFILE=65535
[Install]
WantedBy=multi-user.target启动 Etcd 集群
systemctl daemon-reload && systemctl enable --now etcd
systemctl status etcd查看 Etcd 集群状态
etcdctl --endpoints="10.0.0.41:2379,10.0.0.42:2379,10.0.0.43:2379" --cacert=/caofacan/certs/etcd/etcd-ca.pem --cert=/caofacan/certs/etcd/etcd-server.pem --key=/caofacan/certs/etcd/etcd-server-key.pem endpoint status --write-out=table输出示例：
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 10.0.0.41:2379 | 9378902f41df91e9 | 3.5.21 | 4.9 MB | true | false | 30 | 58023 | 58023 | |
| 10.0.0.42:2379 | 18f972748ec1bd96 | 3.5.21 | 5.0 MB | false | false | 30 | 58023 | 58023 | |
| 10.0.0.43:2379 | a3dfd2d37c461ee9 | 3.5.21 | 4.9 MB | false | false | 30 | 58023 | 58023 | |
+----------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+2.K8S 集群各主机环境准备
下载并解压 K8S 程序包
wget https://dl.k8s.io/v1.32.4/kubernetes-server-linux-amd64.tar.gz
tar xf kubernetes-server-linux-amd64-v1.32.4.tar.gz --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}查看 kubelet 版本
kubelet --version分发软件包到其他节点
for i in `ls -1 /usr/local/bin/kube*`; do data_rsync.sh $i; done所有节点安装常用软件包
apt -y install bind9-utils expect rsync jq psmisc net-tools lvm2 vim unzip rename treenode-exporter41 节点免密钥登录集群并同步数据
-
配置免密码登录其他节点
cat > password_free_login.sh <<'EOF' #!/bin/bash # auther: Jason Yin # 创建密钥对 ssh-keygen -t rsa -P "" -f /root/.ssh/id_rsa -q # 声明你服务器密码，建议所有节点的密码均一致，否则该脚本需要再次进行优化 export mypasswd=1 # 定义主机列表 k8s_host_list=(node-exporter41 node-exporter42 node-exporter43) # 配置免密登录，利用expect工具免交互输入 for i in ${k8s_host_list[@]};do expect -c " spawn ssh-copy-id -i /root/.ssh/id_rsa.pub root@$i expect { "*yes/no*" {send "yes\r"; exp_continue} "*password*" {send "$mypasswd\r"; exp_continue} }" done EOF bash password_free_login.sh -
编写同步脚本
cat > /usr/local/sbin/data_rsync.sh <<'EOF' #!/bin/bash if [ $# -lt 1 ];then echo "Usage: $0 /path/to/file(绝对路径) [mode: m|w]" exit fi if [ ! -e $1 ];then echo "[ $1 ] dir or file not find!" exit fi fullpath=`dirname $1` basename=`basename $1` cd $fullpath case $2 in WORKER_NODE|w) K8S_NODE=(node-exporter42 node-exporter43) ;; MASTER_NODE|m) K8S_NODE=(node-exporter42 node-exporter43) ;; *) K8S_NODE=(node-exporter42 node-exporter43) ;; esac for host in ${K8S_NODE[@]};do tput setaf 2 echo ===== rsyncing ${host}: $basename ===== tput setaf 7 rsync -az $basename `whoami`@${host}:$fullpath if [ $? -eq 0 ];then echo "命令执行成功!" fi done EOF chmod +x /usr/local/sbin/data_rsync.sh data_rsync.sh /etc/hosts
所有节点 Linux 基础环境优化
systemctl disable --now NetworkManager ufw
swapoff -a && sysctl -w vm.swappiness=0
sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab
free -h
ln -svf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
cat >> /etc/security/limits.conf <<'EOF'
* soft nofile 655360
* hard nofile 131072
* soft nproc 655350
* hard nproc 655350
* soft memlock unlimited
* hard memlock unlimited
EOF
sed -i 's@#UseDNS yes@UseDNS no@g' /etc/ssh/sshd_config
sed -i 's@^GSSAPIAuthentication yes@GSSAPIAuthentication no@g' /etc/ssh/sshd_config
cat > /etc/sysctl.d/k8s.conf <<'EOF'
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv6.conf.all.disable_ipv6 = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system
cat <> ~/.bashrc
PS1='[\[\e[34;1m\]\u@\[\e[0m\]\[\e[32;1m\]\H\[\e[0m\]\[\e[31;1m\] \W\[\e[0m\]]# '
EOF
source ~/.bashrc 所有节点安装 ipvsadm 以实现 kube-proxy 的负载均衡
apt -y install ipvsadm ipset sysstat conntrack
cat > /etc/modules-load.d/ipvs.conf << 'EOF'
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
br_netfilter
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF验证模块是否加载成功
reboot
lsmod | grep --color=auto -e ip_vs -e nf_conntrack -e br_netfilter部署 Containerd 运行时
# 这里我自己写了一个自动部署containerd脚本
tar xf oldboyedu-autoinstall-containerd-v1.6.36.tar.gz
./install-containerd.sh i检查 Containerd 版本：
ctr version3.生成 K8S 组件相关证书
生成 K8S 组件相关证书
创建证书的 CSR 文件
mkdir -pv /caofacan/certs/pki && cd /caofacan/certs/pki
cat > k8s-ca-csr.json <生成 K8S 证书
mkdir -pv /caofacan/certs/kubernetes/
cfssl gencert -initca k8s-ca-csr.json | cfssljson -bare /caofacan/certs/kubernetes/k8s-ca生成 K8S 证书的有效期为 100 年
cat > k8s-ca-config.json <生成 apiserver 证书的 CSR 文件
cat > apiserver-csr.json <基于自建 ca 证书生成 apiServer 的证书文件
cfssl gencert \
-ca=/caofacan/certs/kubernetes/k8s-ca.pem \
-ca-key=/caofacan/certs/kubernetes/k8s-ca-key.pem \
-config=k8s-ca-config.json \
--hostname=10.200.0.1,10.0.0.240,kubernetes,kubernetes.default,kubernetes.default.svc,caofacan.com,kubernetes.default.svc.caofacan.com,10.0.0.41,10.0.0.42,10.0.0.43 \
--profile=kubernetes \
apiserver-csr.json | cfssljson -bare /caofacan/certs/kubernetes/apiserver生成聚合证书的用于自建 ca 的 CSR 文件
cat > front-proxy-ca-csr.json <生成聚合证书的自建 ca 证书
cfssl gencert -initca front-proxy-ca-csr.json | cfssljson -bare /caofacan/certs/kubernetes/front-proxy-ca生成聚合证书的用于客户端的 CSR 文件
cat > front-proxy-client-csr.json <基于聚合证书的自建 ca 证书签发聚合证书的客户端证书
cfssl gencert \
-ca=/caofacan/certs/kubernetes/front-proxy-ca.pem \
-ca-key=/caofacan/certs/kubernetes/front-proxy-ca-key.pem \
-config=k8s-ca-config.json \
-profile=kubernetes \
front-proxy-client-csr.json | cfssljson -bare /caofacan/certs/kubernetes/front-proxy-client4.生成 controller-manager 证书及 kubeconfig 文件
-
生成 controller-manager 的 CSR 文件
cat > controller-manager-csr.json < -
生成 controller-manager 证书文件
cfssl gencert \ -ca=/caofacan/certs/kubernetes/k8s-ca.pem \ -ca-key=/caofacan/certs/kubernetes/k8s-ca-key.pem \ -config=k8s-ca-config.json \ -profile=kubernetes \ controller-manager-csr.json | cfssljson -bare /caofacan/certs/kubernetes/controller-manager -
创建 kubeconfig 目录
mkdir -pv /caofacan/certs/kubeconfig -
设置集群
kubectl config set-cluster yinzhengjie-k8s \ --certificate-authority=/caofacan/certs/kubernetes/k8s-ca.pem \ --embed-certs=true \ --server=https://10.0.0.240:8443 \ --kubeconfig=/caofacan/certs/kubeconfig/kube-controller-manager.kubeconfig -
设置用户项
kubectl config set-credentials system:kube-controller-manager \ --client-certificate=/caofacan/certs/kubernetes/controller-manager.pem \ --client-key=/caofacan/certs/kubernetes/controller-manager-key.pem \ --embed-certs=true \ --kubeconfig=/caofacan/certs/kubeconfig/kube-controller-manager.kubeconfig -
设置上下文环境
kubectl config set-context system:kube-controller-manager@kubernetes \ --cluster=yinzhengjie-k8s \ --user=system:kube-controller-manager \ --kubeconfig=/caofacan/certs/kubeconfig/kube-controller-manager.kubeconfig -
使用默认上下文
kubectl config use-context system:kube-controller-manager@kubernetes \ --kubeconfig=/caofacan/certs/kubeconfig/kube-controller-manager.kubeconfig
5.生成 scheduler 证书及 kubeconfig 文件
-
生成 scheduler 的 CSR 文件
cat > scheduler-csr.json < -
生成 scheduler 证书文件
cfssl gencert \ -ca=/caofacan/certs/kubernetes/k8s-ca.pem \ -ca-key=/caofacan/certs/kubernetes/k8s-ca-key.pem \ -config=k8s-ca-config.json \ -profile=kubernetes \ scheduler-csr.json | cfssljson -bare /caofacan/certs/kubernetes/scheduler -
设置集群
kubectl config set-cluster yinzhengjie-k8s \ --certificate-authority=/caofacan/certs/kubernetes/k8s-ca.pem \ --embed-certs=true \ --server=https://10.0.0.240:8443 \ --kubeconfig=/caofacan/certs/kubeconfig/kube-scheduler.kubeconfig -
设置用户项
kubectl config set-credentials system:kube-scheduler \ --client-certificate=/caofacan/certs/kubernetes/scheduler.pem \ --client-key=/caofacan/certs/kubernetes/scheduler-key.pem \ --embed-certs=true \ --kubeconfig=/caofacan/certs/kubeconfig/kube-scheduler.kubeconfig -
设置上下文环境
kubectl config set-context system:kube-scheduler@kubernetes \ --cluster=yinzhengjie-k8s \ --user=system:kube-scheduler \ --kubeconfig=/caofacan/certs/kubeconfig/kube-scheduler.kubeconfig -
使用默认上下文
kubectl config use-context system:kube-scheduler@kubernetes \ --kubeconfig=/caofacan/certs/kubeconfig/kube-scheduler.kubeconfig
6.配置 K8S 集群管理员证书及 kubeconfig 文件
-
生成管理员的 CSR 文件
cat > admin-csr.json < -
生成 K8S 集群管理员证书
cfssl gencert \ -ca=/caofacan/certs/kubernetes/k8s-ca.pem \ -ca-key=/caofacan/certs/kubernetes/k8s-ca-key.pem \ -config=k8s-ca-config.json \ -profile=kubernetes \ admin-csr.json | cfssljson -bare /caofacan/certs/kubernetes/admin -
设置集群
kubectl config set-cluster yinzhengjie-k8s \ --certificate-authority=/caofacan/certs/kubernetes/k8s-ca.pem \ --embed-certs=true \ --server=https://10.0.0.240:8443 \ --kubeconfig=/caofacan/certs/kubeconfig/kube-admin.kubeconfig -
设置用户项
kubectl config set-credentials kube-admin \ --client-certificate=/caofacan/certs/kubernetes/admin.pem \ --client-key=/caofacan/certs/kubernetes/admin-key.pem \ --embed-certs=true \ --kubeconfig=/caofacan/certs/kubeconfig/kube-admin.kubeconfig -
设置上下文环境
kubectl config set-context kube-admin@kubernetes \ --cluster=yinzhengjie-k8s \ --user=kube-admin \ --kubeconfig=/caofacan/certs/kubeconfig/kube-admin.kubeconfig -
使用默认上下文
kubectl config use-context kube-admin@kubernetes \ --kubeconfig=/caofacan/certs/kubeconfig/kube-admin.kubeconfig
7.创建 ServiceAccount
openssl genrsa -out /caofacan/certs/kubernetes/sa.key 2048
openssl rsa -in /caofacan/certs/kubernetes/sa.key -pubout -out /caofacan/certs/kubernetes/sa.pub-
将 K8S 组件证书拷贝到其他两个 master 节点
data_rsync.sh /caofacan/certs/kubeconfig/
data_rsync.sh /caofacan/certs/kubernetes/8.高可用组件 haproxy + keepalived 安装及验证
所有 master 节点安装高可用组件
apt-get -y install keepalived haproxy配置 haproxy
-
备份配置文件
cp /etc/haproxy/haproxy.cfg{,`date +%F`} -
所有节点的配置文件内容相同
cat > /etc/haproxy/haproxy.cfg <<'EOF' global maxconn 2000 ulimit-n 16384 log 127.0.0.1 local0 err stats timeout 30s defaults log global mode http option httplog timeout connect 5000 timeout client 50000 timeout server 50000 timeout http-request 15s timeout http-keep-alive 15s frontend monitor-haproxy bind *:9999 mode http option httplog monitor-uri /ruok frontend yinzhengjie-k8s bind 0.0.0.0:8443 bind 127.0.0.1:8443 mode tcp option tcplog tcp-request inspect-delay 5s default_backend yinzhengjie-k8s backend yinzhengjie-k8s mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server node-exporter41 10.0.0.41:6443 check server node-exporter42 10.0.0.42:6443 check server node-exporter43 10.0.0.43:6443 check EOF
配置 keepalived
-
node-exporter41 节点创建配置文件
cat > /etc/keepalived/keepalived.conf <<'EOF' ! Configuration File for keepalived global_defs { router_id 10.0.0.41 } vrrp_script chk_nginx { script "/etc/keepalived/check_port.sh 8443" interval 2 weight -20 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 251 priority 100 advert_int 1 mcast_src_ip 10.0.0.41 nopreempt authentication { auth_type PASS auth_pass yinzhengjie_k8s } track_script { chk_nginx } virtual_ipaddress { 10.0.0.240 } } EOF -
node-exporter42 和 node-exporter43 节点创建配置文件：类似 node-exporter41，需修改对应 IP。
-
所有 keepalived 节点创建健康检查脚本
cat > /etc/keepalived/check_port.sh <<'EOF' #!/bin/bash CHK_PORT=$1 if [ -n "$CHK_PORT" ];then PORT_PROCESS=`ss -lt|grep $CHK_PORT|wc -l` if [ $PORT_PROCESS -eq 0 ];then echo "Port $CHK_PORT Is Not Used,End." systemctl stop keepalived fi else echo "Check Port Cant Be Empty!" fi EOF chmod +x /etc/keepalived/check_port.sh
验证 haproxy 服务
-
所有节点启动 haproxy 服务
systemctl enable --now haproxy systemctl restart haproxy systemctl status haproxy ss -ntl | egrep "8443|9999" -
基于 webUI 进行验证
curl http://10.0.0.41:9999/ruok curl http://10.0.0.42:9999/ruok curl http://10.0.0.43:9999/ruok
启动 keepalived 服务并验证
-
所有节点启动 keepalived 服务
systemctl daemon-reload systemctl enable --now keepalived systemctl status keepalived -
验证服务是否正常
ip a -
基于 telnet 验证 haproxy 是否正常
telnet 10.0.0.240 8443 ping 10.0.0.240 -c 3
9.部署 ApiServer 组件
node-exporter41 节点启动 ApiServer
-
创建 node-exporter41 节点的配置文件
cat > /usr/lib/systemd/system/kube-apiserver.service << 'EOF' [Unit] Description=Jason Yin's Kubernetes API Server Documentation=https://github.com/kubernetes/kubernetes After=network.target [Service] ExecStart=/usr/local/bin/kube-apiserver \ --v=2 \ --bind-address=0.0.0.0 \ --secure-port=6443 \ --allow_privileged=true \ --advertise-address=10.0.0.41 \ --service-cluster-ip-range=10.200.0.0/16 \ --service-node-port-range=3000-50000 \ --etcd-servers=https://10.0.0.41:2379,https://10.0.0.42:2379,https://10.0.0.43:2379 \ --etcd-cafile=/caofacan/certs/etcd/etcd-ca.pem \ --etcd-certfile=/caofacan/certs/etcd/etcd-server.pem \ --etcd-keyfile=/caofacan/certs/etcd/etcd-server-key.pem \ --client-ca-file=/caofacan/certs/kubernetes/k8s-ca.pem \ --tls-cert-file=/caofacan/certs/kubernetes/apiserver.pem \ --tls-private-key-file=/caofacan/certs/kubernetes/apiserver-key.pem \ --kubelet-client-certificate=/caofacan/certs/kubernetes/apiserver.pem \ --kubelet-client-key=/caofacan/certs/kubernetes/apiserver-key.pem \ --service-account-key-file=/caofacan/certs/kubernetes/sa.pub \ --service-account-signing-key-file=/caofacan/certs/kubernetes/sa.key \ --service-account-issuer=https://kubernetes.default.svc.caofacan.com \ --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname \ --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota \ --authorization-mode=Node,RBAC \ --enable-bootstrap-token-auth=true \ --requestheader-client-ca-file=/caofacan/certs/kubernetes/front-proxy-ca.pem \ --proxy-client-cert-file=/caofacan/certs/kubernetes/front-proxy-client.pem \ --proxy-client-key-file=/caofacan/certs/kubernetes/front-proxy-client-key.pem \ --requestheader-allowed-names=aggregator \ --requestheader-group-headers=X-Remote-Group \ --requestheader-extra-headers-prefix=X-Remote-Extra- \ --requestheader-username-headers=X-Remote-User Restart=on-failure RestartSec=10s LimitNOFILE=65535 [Install] WantedBy=multi-user.target EOF -
启动服务
systemctl daemon-reload && systemctl enable --now kube-apiserver systemctl status kube-apiserver ss -ntl | grep 6443
node-exporter42 和 node-exporter43 节点启动 ApiServer
类似 node-exporter41，需修改对应 IP 和节点名称。
10.部署 ControllerManager 组件
所有节点创建配置文件
cat > /usr/lib/systemd/system/kube-controller-manager.service << 'EOF'
[Unit]
Description=Jason Yin's Kubernetes Controller Manager
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-controller-manager \
--v=2 \
--root-ca-file=/caofacan/certs/kubernetes/k8s-ca.pem \
--cluster-signing-cert-file=/caofacan/certs/kubernetes/k8s-ca.pem \
--cluster-signing-key-file=/caofacan/certs/kubernetes/k8s-ca-key.pem \
--service-account-private-key-file=/caofacan/certs/kubernetes/sa.key \
--kubeconfig=/caofacan/certs/kubeconfig/kube-controller-manager.kubeconfig \
--leader-elect=true \
--use-service-account-credentials=true \
--node-monitor-grace-period=40s \
--node-monitor-period=5s \
--controllers=*,bootstrapsigner,tokencleaner \
--allocate-node-cidrs=true \
--cluster-cidr=10.100.0.0/16 \
--requestheader-client-ca-file=/caofacan/certs/kubernetes.front-proxy-ca.pem \
--node-cidr-mask-size=24
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
EOF启动 controller-manager 服务
systemctl daemon-reload
systemctl enable --now kube-controller-manager
systemctl status kube-controller-manager
ss -ntl | grep 1025711.部署 Scheduler 组件
所有节点创建配置文件
cat > /usr/lib/systemd/system/kube-scheduler.service <<'EOF'
[Unit]
Description=Jason Yin's Kubernetes Scheduler
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-scheduler \
--v=2 \
--leader-elect=true \
--kubeconfig=/caofacan/certs/kubeconfig/kube-scheduler.kubeconfig
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
EOF启动 scheduler 服务
systemctl daemon-reload
systemctl enable --now kube-scheduler
systemctl status kube-scheduler
ss -ntl | grep 1025912.创建 Bootstrapping 自动颁发 kubelet 证书配置及集群管理证书配置
创建 bootstrap-kubelet.kubeconfig 文件
-
设置集群
kubectl config set-cluster yinzhengjie-k8s \ --certificate-authority=/caofacan/certs/kubernetes/k8s-ca.pem \ --embed-certs=true \ --server=https://10.0.0.240:8443 \ --kubeconfig=/caofacan/certs/kubeconfig/bootstrap-kubelet.kubeconfig -
创建用户
kubectl config set-credentials tls-bootstrap-token-user \ --token=caofacan.jasonyinzhengjie \ --kubeconfig=/caofacan/certs/kubeconfig/bootstrap-kubelet.kubeconfig -
将集群和用户进行绑定
kubectl config set-context tls-bootstrap-token-user@kubernetes \ --cluster=yinzhengjie-k8s \ --user=tls-bootstrap-token-user \ --kubeconfig=/caofacan/certs/kubeconfig/bootstrap-kubelet.kubeconfig -
配置默认的上下文
kubectl config use-context tls-bootstrap-token-user@kubernetes \ --kubeconfig=/caofacan/certs/kubeconfig/bootstrap-kubelet.kubeconfig -
拷贝 kubelet 的 Kubeconfig 文件
data_rsync.sh /caofacan/certs/kubeconfig/bootstrap-kubelet.kubeconfig
13.所有 master 节点拷贝管理证书
-
所有 master 节点拷贝管理员的证书文件
mkdir -p /root/.kube cp /caofacan/certs/kubeconfig/kube-admin.kubeconfig /root/.kube/config -
查看 master 组件
kubectl get cs -
查看集群状态
kubectl cluster-info dump
14.创建 bootstrap-secret 授权
-
创建 bootstrap-secret 文件用于授权
cat > bootstrap-secret.yaml < -
应用 bootstrap-secret 配置文件
kubectl apply -f bootstrap-secret.yaml
15.部署 worker 节点之 kubelet 启动实战
所有节点创建工作目录
mkdir -p /var/lib/kubelet /var/log/kubernetes /etc/systemd/system/kubelet.service.d /etc/kubernetes/manifests/所有节点创建 kubelet 的配置文件
cat > /etc/kubernetes/kubelet-conf.yml <<'EOF'
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: 0.0.0.0
port: 10250
readOnlyPort: 10255
authentication:
anonymous:
enabled: false
webhook:
cacheTTL: 2m0s
enabled: true
x509:
clientCAFile: /caofacan/certs/kubernetes/k8s-ca.pem
authorization:
mode: Webhook
webhook:
cacheAuthorizedTTL: 5m0s
cacheUnauthorizedTTL: 30s
cgroupDriver: systemd
cgroupsPerQOS: true
clusterDNS:
- 10.200.0.254
clusterDomain: caofacan.com
containerLogMaxFiles: 5
containerLogMaxSize: 10Mi
contentType: application/vnd.kubernetes.protobuf
cpuCFSQuota: true
cpuManagerPolicy: none
cpuManagerReconcilePeriod: 10s
enableControllerAttachDetach: true
enableDebuggingHandlers: true
enforceNodeAllocatable:
- pods
eventBurst: 10
eventRecordQPS: 5
evictionHard:
imagefs.available: 15%
memory.available: 100Mi
nodefs.available: 10%
nodefs.inodesFree: 5%
evictionPressureTransitionPeriod: 5m0s
failSwapOn: true
fileCheckFrequency: 20s
hairpinMode: promiscuous-bridge
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 20s
imageGCHighThresholdPercent: 85
imageGCLowThresholdPercent: 80
imageMinimumGCAge: 2m0s
iptablesDropBit: 15
iptablesMasqueradeBit: 14
kubeAPIBurst: 10
kubeAPIQPS: 5
makeIPTablesUtilChains: true
maxOpenFiles: 1000000
maxPods: 110
nodeStatusUpdateFrequency: 10s
oomScoreAdj: -999
podPidsLimit: -1
registryBurst: 10
registryPullQPS: 5
resolvConf: /etc/kubernetes/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 2m0s
serializeImagePulls: true
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 4h0m0s
syncFrequency: 1m0s
volumeStatsAggPeriod: 1m0s
EOF所有节点配置 kubelet service
cat > /usr/lib/systemd/system/kubelet.service <<'EOF'
[Unit]
Description=JasonYin's Kubernetes Kubelet
Documentation=https://github.com/kubernetes/kubernetes
After=containerd.service
Requires=containerd.service
[Service]
ExecStart=/usr/local/bin/kubelet
Restart=always
StartLimitInterval=0
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF所有节点配置 kubelet service 的配置文件
cat > /etc/systemd/system/kubelet.service.d/10-kubelet.conf <<'EOF'
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/caofacan/certs/kubeconfig/bootstrap-kubelet.kubeconfig --kubeconfig=/caofacan/certs/kubeconfig/kubelet.kubeconfig"
Environment="KUBELET_CONFIG_ARGS=--config=/etc/kubernetes/kubelet-conf.yml"
Environment="KUBELET_SYSTEM_ARGS=--container-runtime-endpoint=unix:///run/containerd/containerd.sock"
Environment="KUBELET_EXTRA_ARGS=--node-labels=node.kubernetes.io/node='' "
ExecStart=
ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_SYSTEM_ARGS $KUBELET_EXTRA_ARGS
EOF启动所有节点 kubelet
systemctl daemon-reload
systemctl enable --now kubelet
systemctl status kubelet在所有 master 节点上查看 nodes 信息
kubectl get nodes -o wide查看 csr 用户客户端的证书请求
kubectl get csr16.部署 worker 节点之 kube-proxy 服务
生成 kube-proxy 的 csr 文件
cat > kube-proxy-csr.json <创建 kube-proxy 需要的证书文件
cfssl gencert \
-ca=/caofacan/certs/kubernetes/k8s-ca.pem \
-ca-key=/caofacan/certs/kubernetes/k8s-ca-key.pem \
-config=k8s-ca-config.json \
-profile=kubernetes \
kube-proxy-csr.json | cfssljson -bare /caofacan/certs/kubernetes/kube-proxy生成 Kubeconfig 文件
-
设置集群
kubectl config set-cluster yinzhengjie-k8s \ --certificate-authority=/caofacan/certs/kubernetes/k8s-ca.pem \ --embed-certs=true \ --server=https://10.0.0.240:8443 \ --kubeconfig=/caofacan/certs/kubeconfig/kube-proxy.kubeconfig -
设置用户项
kubectl config set-credentials system:kube-proxy \ --client-certificate=/caofacan/certs/kubernetes/kube-proxy.pem \ --client-key=/caofacan/certs/kubernetes/kube-proxy-key.pem \ --embed-certs=true \ --kubeconfig=/caofacan/certs/kubeconfig/kube-proxy.kubeconfig -
设置上下文环境
kubectl config set-context kube-proxy@kubernetes \ --cluster=yinzhengjie-k8s \ --user=system:kube-proxy \ --kubeconfig=/caofacan/certs/kubeconfig/kube-proxy.kubeconfig -
使用默认上下文
kubectl config use-context kube-proxy@kubernetes \ --kubeconfig=/caofacan/certs/kubeconfig/kube-proxy.kubeconfig
将 Kubeconfig 文件同步到所有的 worker 节点
data_rsync.sh /caofacan/certs/kubeconfig/kube-proxy.kubeconfig所有节点创建 kube-proxy.conf 配置文件
cat > /etc/kubernetes/kube-proxy.yml << EOF
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
bindAddress: 0.0.0.0
metricsBindAddress: 127.0.0.1:10249
clientConnection:
acceptConnection: ""
burst: 10
contentType: application/vnd.kubernetes.protobuf
kubeconfig: /caofacan/certs/kubeconfig/kube-proxy.kubeconfig
qps: 5
clusterCIDR: 10.100.0.0/16
configSyncPeriod: 15m0s
conntrack:
max: null
maxPerCore: 32768
min: 131072
tcpCloseWaitTimeout: 1h0m0s
tcpEstablishedTimeout: 24h0m0s
enableProfiling: false
healthzBindAddress: 0.0.0.0:10256
hostnameOverride: ""
iptables:
masqueradeAll: false
masqueradeBit: 14
minSyncPeriod: 0s
ipvs:
masqueradeAll: true
minSyncPeriod: 5s
scheduler: "rr"
syncPeriod: 30s
mode: "ipvs"
nodeProtAddress: null
oomScoreAdj: -999
portRange: ""
udpIdelTimeout: 250ms
EOF所有节点使用 systemd 管理 kube-proxy
cat > /usr/lib/systemd/system/kube-proxy.service << EOF
[Unit]
Description=Jason Yin's Kubernetes Proxy
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-proxy \
--config=/etc/kubernetes/kube-proxy.yml \
--v=2
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF所有节点启动 kube-proxy
systemctl daemon-reload && systemctl enable --now kube-proxy
systemctl status kube-proxy
ss -ntl |egrep "10256|10249"17.附加组件部署CNI之flannel
参考链接:
https://github.com/flannel-io/flannel?tab=readme-ov-file#deploying-flannel-manually
1.下载资源清单
2.修改Flannel的资源清单
[root@node-exporter41 ~]# grep 16 kube-flannel.yml
"Network": "10.244.0.0/16",
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# sed -i '/16/s#244#100#' kube-flannel.yml
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# grep 16 kube-flannel.yml
"Network": "10.100.0.0/16",
[root@node-exporter41 ~]#
[root@node-exporter41 ~]#
3.所有节点安装CNI通用插件
wget https://github.com/containernetworking/plugins/releases/download/v1.6.2/cni-plugins-linux-amd64-v1.6.2.tgz
tar xf cni-plugins-linux-amd64-v1.6.2.tgz -C /opt/cni/bin/
[root@node-exporter41 ~]# ll /opt/cni/bin/
total 89780
drwxr-xr-x 2 root root 4096 Jan 7 00:12 ./
drwxr-xr-x 3 root root 4096 Apr 23 14:43 ../
-rwxr-xr-x 1 root root 4655178 Jan 7 00:12 bandwidth*
-rwxr-xr-x 1 root root 5287212 Jan 7 00:12 bridge*
-rwxr-xr-x 1 root root 12762814 Jan 7 00:12 dhcp*
-rwxr-xr-x 1 root root 4847854 Jan 7 00:12 dummy*
-rwxr-xr-x 1 root root 5315134 Jan 7 00:12 firewall*
-rwxr-xr-x 1 root root 4792010 Jan 7 00:12 host-device*
-rwxr-xr-x 1 root root 4060355 Jan 7 00:12 host-local*
-rwxr-xr-x 1 root root 4870719 Jan 7 00:12 ipvlan*
-rw-r--r-- 1 root root 11357 Jan 7 00:12 LICENSE
-rwxr-xr-x 1 root root 4114939 Jan 7 00:12 loopback*
-rwxr-xr-x 1 root root 4903324 Jan 7 00:12 macvlan*
-rwxr-xr-x 1 root root 4713429 Jan 7 00:12 portmap*
-rwxr-xr-x 1 root root 5076613 Jan 7 00:12 ptp*
-rw-r--r-- 1 root root 2343 Jan 7 00:12 README.md
-rwxr-xr-x 1 root root 4333422 Jan 7 00:12 sbr*
-rwxr-xr-x 1 root root 3651755 Jan 7 00:12 static*
-rwxr-xr-x 1 root root 4928874 Jan 7 00:12 tap*
-rwxr-xr-x 1 root root 4208424 Jan 7 00:12 tuning*
-rwxr-xr-x 1 root root 4868252 Jan 7 00:12 vlan*
-rwxr-xr-x 1 root root 4488658 Jan 7 00:12 vrf*
[root@node-exporter41 ~]#
[root@node-exporter42 ~]# ll /opt/cni/bin/
total 89780
drwxr-xr-x 2 root root 4096 Apr 23 14:56 ./
drwxr-xr-x 3 root root 4096 Apr 23 14:43 ../
-rwxr-xr-x 1 root root 4655178 Jan 7 00:12 bandwidth*
-rwxr-xr-x 1 root root 5287212 Jan 7 00:12 bridge*
-rwxr-xr-x 1 root root 12762814 Jan 7 00:12 dhcp*
-rwxr-xr-x 1 root root 4847854 Jan 7 00:12 dummy*
-rwxr-xr-x 1 root root 5315134 Jan 7 00:12 firewall*
-rwxr-xr-x 1 root root 4792010 Jan 7 00:12 host-device*
-rwxr-xr-x 1 root root 4060355 Jan 7 00:12 host-local*
-rwxr-xr-x 1 root root 4870719 Jan 7 00:12 ipvlan*
-rw-r--r-- 1 root root 11357 Jan 7 00:12 LICENSE
-rwxr-xr-x 1 root root 4114939 Jan 7 00:12 loopback*
-rwxr-xr-x 1 root root 4903324 Jan 7 00:12 macvlan*
-rwxr-xr-x 1 root root 4713429 Jan 7 00:12 portmap*
-rwxr-xr-x 1 root root 5076613 Jan 7 00:12 ptp*
-rw-r--r-- 1 root root 2343 Jan 7 00:12 README.md
-rwxr-xr-x 1 root root 4333422 Jan 7 00:12 sbr*
-rwxr-xr-x 1 root root 3651755 Jan 7 00:12 static*
-rwxr-xr-x 1 root root 4928874 Jan 7 00:12 tap*
-rwxr-xr-x 1 root root 4208424 Jan 7 00:12 tuning*
-rwxr-xr-x 1 root root 4868252 Jan 7 00:12 vlan*
-rwxr-xr-x 1 root root 4488658 Jan 7 00:12 vrf*
[root@node-exporter42 ~]#
[root@node-exporter42 ~]#
[root@node-exporter43 ~]# ll /opt/cni/bin/
total 89780
drwxr-xr-x 2 root root 4096 Apr 23 14:56 ./
drwxr-xr-x 3 root root 4096 Apr 23 14:43 ../
-rwxr-xr-x 1 root root 4655178 Jan 7 00:12 bandwidth*
-rwxr-xr-x 1 root root 5287212 Jan 7 00:12 bridge*
-rwxr-xr-x 1 root root 12762814 Jan 7 00:12 dhcp*
-rwxr-xr-x 1 root root 4847854 Jan 7 00:12 dummy*
-rwxr-xr-x 1 root root 5315134 Jan 7 00:12 firewall*
-rwxr-xr-x 1 root root 4792010 Jan 7 00:12 host-device*
-rwxr-xr-x 1 root root 4060355 Jan 7 00:12 host-local*
-rwxr-xr-x 1 root root 4870719 Jan 7 00:12 ipvlan*
-rw-r--r-- 1 root root 11357 Jan 7 00:12 LICENSE
-rwxr-xr-x 1 root root 4114939 Jan 7 00:12 loopback*
-rwxr-xr-x 1 root root 4903324 Jan 7 00:12 macvlan*
-rwxr-xr-x 1 root root 4713429 Jan 7 00:12 portmap*
-rwxr-xr-x 1 root root 5076613 Jan 7 00:12 ptp*
-rw-r--r-- 1 root root 2343 Jan 7 00:12 README.md
-rwxr-xr-x 1 root root 4333422 Jan 7 00:12 sbr*
-rwxr-xr-x 1 root root 3651755 Jan 7 00:12 static*
-rwxr-xr-x 1 root root 4928874 Jan 7 00:12 tap*
-rwxr-xr-x 1 root root 4208424 Jan 7 00:12 tuning*
-rwxr-xr-x 1 root root 4868252 Jan 7 00:12 vlan*
-rwxr-xr-x 1 root root 4488658 Jan 7 00:12 vrf*
[root@node-exporter43 ~]#
4.部署服务
[root@node-exporter41 ~]# kubectl apply -f kube-flannel.yml
namespace/kube-flannel created
serviceaccount/flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
[root@node-exporter41 ~]#
5.检查配置
[root@node-exporter41 ~]# kubectl get pods -o wide -A
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel kube-flannel-ds-8pt5n 1/1 Running 0 5s 10.0.0.41 node-exporter41
kube-flannel kube-flannel-ds-gwhpb 1/1 Running 0 5s 10.0.0.43 node-exporter43
kube-flannel kube-flannel-ds-mtmxt 1/1 Running 0 5s 10.0.0.42 node-exporter42
[root@node-exporter41 ~]#
6.验证CNI网络插件是否正常工作
[root@node-exporter41 ~]# cat > network-cni-test.yaml < node-exporter42
xiuxian-v2 0/1 ContainerCreating 0 4s node-exporter43
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
xiuxian-v1 1/1 Running 0 9s 10.100.3.2 node-exporter42
xiuxian-v2 1/1 Running 0 9s 10.100.0.2 node-exporter43
[root@node-exporter41 ~]#
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# curl 10.100.3.2
yinzhengjie apps v1
凡人修仙传 v1
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# curl 10.100.0.2
yinzhengjie apps v2
凡人修仙传 v2
[root@node-exporter41 ~]#
7.开启自动补全功能
kubectl completion bash > ~/.kube/completion.bash.inc
echo "source '$HOME/.kube/completion.bash.inc'" >> $HOME/.bashrc
source $HOME/.bashrc
18.CoreDNS部署及故障排查
1 下载资源清单
参考链接:
https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/dns/coredns
[root@node-exporter41 ~]# wget http://192.168.16.253/Resources/Kubernetes/Add-ons/CoreDNS/coredns.yaml.base
2 修改资源清单模板的关键字段
[root@node-exporter41 ~]# sed -i '/__DNS__DOMAIN__/s#__DNS__DOMAIN__#oldboyedu.com#' coredns.yaml.base
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# sed -i '/__DNS__MEMORY__LIMIT__/s#__DNS__MEMORY__LIMIT__#200Mi#' coredns.yaml.base
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# sed -i '/__DNS__SERVER__/s#__DNS__SERVER__#10.200.0.254#' coredns.yaml.base
[root@node-exporter41 ~]#
相关字段说明:
__DNS__DOMAIN__
DNS自定义域名，要和你实际的K8S域名对应上。
__DNS__MEMORY__LIMIT__
CoreDNS组件的内存限制。
__DNS__SERVER__
DNS服务器的svc的CLusterIP地址。
3 kubelet指定Pod的DNS服务器
3.1 准备解析的配置文件
[root@node-exporter41 ~]# cat > /etc/kubernetes/resolv.conf <
[root@node-exporter41 ~]#
温馨提示:
如果镜像下载失败，可以手动导入。操作如下:
wget http://192.168.16.253/Resources/Kubernetes/Add-ons/CoreDNS/oldboyedu-coredns-v1.12.0.tar.gz
ctr -n k8s.io i import oldboyedu-coredns-v1.12.0.tar.gz
5.验证DNS服务
[root@node-exporter41 ~]# kubectl get svc -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 10.200.0.1 443/TCP 3h56m
kube-system kube-dns ClusterIP 10.200.0.254 53/UDP,53/TCP,9153/TCP 15m
[root@node-exporter41 ~]#
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# dig @10.200.0.254 kube-dns.kube-system.svc.oldboyedu.com +short
10.200.0.254
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# dig @10.200.0.254 kubernetes.default.svc.oldboyedu.com +short
10.200.0.1
[root@node-exporter41 ~]#
6.部署Pod验证默认的DNS服务器
[root@node-exporter41 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
xiuxian-v1 1/1 Running 0 19s 10.100.3.5 node-exporter42
xiuxian-v2 1/1 Running 0 19s 10.100.0.4 node-exporter43
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# kubectl exec -it xiuxian-v1 -- cat /etc/resolv.conf
search default.svc.oldboyedu.com svc.oldboyedu.com oldboyedu.com
nameserver 10.200.0.254
options ndots:5
[root@node-exporter41 ~]#
7.清除Pod环境
[root@node-exporter41 ~]# kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
xiuxian-v1 1/1 Running 0 2m20s 10.100.3.5 node-exporter42
xiuxian-v2 1/1 Running 0 2m20s 10.100.0.4 node-exporter43
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# kubectl delete pods --all
pod "xiuxian-v1" deleted
pod "xiuxian-v2" deleted
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# kubectl get pods -o wide
No resources found in default namespace.
[root@node-exporter41 ~]#
19.验证 K8S 集群高可用
kubectl get no -o wide
查看集群状态：
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
node-exporter41 Ready 8h v1.32.4 10.0.0.41 Ubuntu 22.04.4 LTS 5.15.0-135-generic containerd://1.6.36
node-exporter42 Ready 8h v1.32.4 10.0.0.42 Ubuntu 22.04.4 LTS 5.15.0-134-generic containerd://1.6.36
node-exporter43 Ready 8h v1.32.4 10.0.0.43 Ubuntu 22.04.4 LTS 5.15.0-135-generic containerd://1.6.36
五、附加组件部署
1.Helm
1.安装helm
[root@node-exporter41 ~]# wget https://get.helm.sh/helm-v3.17.3-linux-amd64.tar.gz
[root@node-exporter41 ~]# tar xf helm-v3.17.3-linux-amd64.tar.gz -C /usr/local/bin/ linux-amd64/helm --strip-components=1
[root@node-exporter41 ~]# helm version
version.BuildInfo{Version:"v3.17.3", GitCommit:"e4da49785aa6e6ee2b86efd5dd9e43400318262b", GitTreeState:"clean", GoVersion:"go1.23.7"}
# 同步到其他节点上
[root@node-exporter41 ~]# data_rsync.sh /usr/local/bin/helm
===== rsyncing node-exporter42: helm =====
命令执行成功!
===== rsyncing node-exporter43: helm =====
命令执行成功!
[root@node-exporter41 ~]#
2.所有节点配置helm的自动补全功能
helm completion bash > /etc/bash_completion.d/helm
source /etc/bash_completion.d/helm
echo 'source /etc/bash_completion.d/helm' >> ~/.bashrc
2.Ingress-nginx
github地址:
https://github.com/kubernetes/ingress-nginx
1.添加第三方仓库
[root@node-exporter41 ~]# helm repo add ingress https://kubernetes.github.io/ingress-nginx
"oldboyedu-ingress" has been added to your repositories
[root@node-exporter41 ~]# helm repo list
NAME URL
ingress https://kubernetes.github.io/ingress-nginx
2.下载指定的Chart
[root@node-exporter41 ~]# helm search repo ingress-nginx
NAME CHART VERSION APP VERSION DESCRIPTION
oldboyedu-ingress/ingress-nginx 4.12.1 1.12.1
Ingress controller for Kubernetes using NGINX a...
[root@node-exporter41 ~]# helm pull oldboyedu-ingress/ingress-nginx --version 4.12.1
5.解压软件包并修改配置参数
[root@node-exporter41 ~]# tar xf ingress-nginx-4.12.1.tgz
[root@node-exporter41 ~]# sed -ri '/digest:/s@^@#@' ingress-nginx/values.yaml
[root@node-exporter41 ~]# sed -i '/hostNetwork:/s#false#true#' ingress-nginx/values.yaml
[root@node-exporter41 ~]# sed -i '/dnsPolicy/s#ClusterFirst#ClusterFirstWithHostNet#' ingress-nginx/values.yaml
[root@node-exporter41 ~]# sed -i '/kind/s#Deployment#DaemonSet#' ingress-nginx/values.yaml
[root@node-exporter41 ~]# sed -i '/default:/s#false#true#' ingress-nginx/values.yaml
温馨提示:
注意，记得编辑values.yaml文件，将'admissionWebhooks'功能禁用。即： enabled: false
[root@node-exporter41 ~]# grep -A 10 'admissionWebhooks' ingress-nginx/values.yaml
admissionWebhooks:
..........
enabled: false
6.安装ingress-nginx
[root@node-exporter41 ~]# helm upgrade --install myingress ingress-nginx -n ingress-nginx --create-namespace
Release "myingress" does not exist. Installing it now.
7.验证Ingress-nginx是否安装成功
[root@node-exporter41 ~]# helm -n ingress-nginx list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
myingress ingress-nginx 1 2025-04-23 21:16:21.595928036 +0800 CST deployed ingress-nginx-4.12.1 1.12.1
[root@node-exporter41 ~]# kubectl get ingressclass,deploy,svc,po -n ingress-nginx -o wide
NAME CONTROLLER PARAMETERS AGE
ingressclass.networking.k8s.io/nginx k8s.io/ingress-nginx 2m7s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/myingress-ingress-nginx-controller LoadBalancer 10.200.105.97 80:44150/TCP,443:15236/TCP 2m7s app.kubernetes.io/component=controller,app.kubernetes.io/instance=myingress,app.kubernetes.io/name=ingress-nginx
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/myingress-ingress-nginx-controller-7mnbk 1/1 Running 0 2m7s 10.0.0.42 node-exporter42
pod/myingress-ingress-nginx-controller-hbpnp 1/1 Running 0 2m7s 10.0.0.41 node-exporter41
pod/myingress-ingress-nginx-controller-jkndx 1/1 Running 0 2m7s 10.0.0.43 node-exporter43
8.访问测试，因为我们的配置直接使用宿主机网络即可。
http://10.0.0.41/
http://10.0.0.42/
http://10.0.0.43/
3.metallb
1.下载metallb部署
[root@node-exporter41 ~]# wget https://raw.githubusercontent.com/metallb/metallb/v0.14.9/config/manifests/metallb-native.yaml
[root@node-exporter41 ~]# kubectl apply -f metallb-native.yaml
namespace/metallb-system created
....
2. 查看metallb的状态
[root@node-exporter41 ~]# kubectl get all -o wide -n metallb-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
pod/controller-bb5f47665-tjz84 1/1 Running 0 30s 10.100.3.3 node-exporter43
pod/speaker-h6n9d 1/1 Running 0 30s 10.0.0.42 node-exporter42
pod/speaker-m4kk7 1/1 Running 0 30s 10.0.0.41 node-exporter41
pod/speaker-wxlwj 1/1 Running 0 30s 10.0.0.43 node-exporter43
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
service/metallb-webhook-service ClusterIP 10.200.119.247 443/TCP 30s component=controller
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE CONTAINERS IMAGES SELECTOR
daemonset.apps/speaker 3 3 3 3 3 kubernetes.io/os=linux 30s speaker quay.io/metallb/speaker:v0.14.9 app=metallb,component=speaker
NAME READY UP-TO-DATE AVAILABLE AGE CONTAINERS IMAGES SELECTOR
deployment.apps/controller 1/1 1 1 30s controller quay.io/metallb/controller:v0.14.9 app=metallb,component=controller
NAME DESIRED CURRENT READY AGE CONTAINERS IMAGES SELECTOR
replicaset.apps/controller-bb5f47665 1 1 1 30s controller quay.io/metallb/controller:v0.14.9 app=metallb,component=controller,pod-template-hash=bb5f47665
3.创建MetalLB地址池
[root@node-exporter41 ~]# cat > metallb-ip-pool.yaml < 443/TCP 9h
ingress-nginx myingress-ingress-nginx-controller LoadBalancer 10.200.105.97 10.0.0.150 80:44150/TCP,443:15236/TCP 5m49s
4.metrics-server
部署文档
https://github.com/kubernetes-sigs/metrics-server
1 下载资源清单
[root@node-exporter41 ~]# wget https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/high-availability-1.21+.yaml
2.编辑配置文件
[root@node-exporter41 ~]# vim high-availability-1.21+.yaml
...
114 apiVersion: apps/v1
115 kind: Deployment
116 metadata:
...
144 - args:
145 - --kubelet-insecure-tls # 不要验证Kubelets提供的服务证书的CA。不配置则会报错x509。
...
### 验证所有节点是否都配置了'--requestheader-allowed-names=front-proxy-client'参数
vim /lib/systemd/system/kube-apiserver.service
[Unit]
Description=Jason Yin's Kubernetes API Server
Documentation=https://github.com/kubernetes/kubernetes
After=network.target
[Service]
ExecStart=/usr/local/bin/kube-apiserver \
--requestheader-allowed-names=front-proxy-client \
--v=2 \
2.如果没有配置参数添加后重启服务即可。
systemctl daemon-reload && systemctl restart kube-apiserver
ss -ntl |grep 6443
LISTEN 0 16384 *:6443 *:*
ss -ntl |grep 8443
LISTEN 0 2000 127.0.0.1:8443 0.0.0.0:*
LISTEN 0 2000 0.0.0.0:8443 0.0.0.0:*
3.部署metrics-server
[root@node-exporter41 ~]# kubectl apply -f high-availability-1.21+.yaml
4.验证测试
[root@node-exporter41 ~]# kubectl get pods -o wide -n kube-system -l k8s-app=metrics-server
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
metrics-server-9c85f7647-8lfrz 1/1 Running 0 28s 10.100.1.3 node-exporter42
metrics-server-9c85f7647-kpzjz 1/1 Running 0 28s 10.100.3.4 node-exporter43
[root@node-exporter41 ~]# kubectl top no
NAME CPU(cores) CPU(%) MEMORY(bytes) MEMORY(%)
node-exporter41 109m 5% 1064Mi 18%
node-exporter42 116m 5% 866Mi 14%
node-exporter43 218m 10% 1169Mi 20%
5.基于helm部署Dashboard
参考链接:
https://github.com/kubernetes/dashboard
1.添加Dashboard的仓库地址
[root@node-exporter41 ~]# helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard/
"kubernetes-dashboard" has been added to your repositories
[root@node-exporter41 ~]# helm repo list
NAME URL
oldboyedu-ingress https://kubernetes.github.io/ingress-nginx
kubernetes-dashboard https://kubernetes.github.io/dashboard/
2.安装Dashboard
[root@node-exporter41 ~]# helm upgrade --install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard
[root@node-exporter41 ~]# tar xf kubernetes-dashboard-7.12.0.tgz
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# ll kubernetes-dashboard
total 56
drwxr-xr-x 4 root root 4096 Apr 23 21:46 ./
drwx------ 10 root root 4096 Apr 23 21:46 ../
-rw-r--r-- 1 root root 497 Apr 16 23:19 Chart.lock
drwxr-xr-x 6 root root 4096 Apr 23 21:46 charts/
-rw-r--r-- 1 root root 982 Apr 16 23:19 Chart.yaml
-rw-r--r-- 1 root root 948 Apr 16 23:19 .helmignore
-rw-r--r-- 1 root root 8209 Apr 16 23:19 README.md
drwxr-xr-x 10 root root 4096 Apr 23 21:46 templates/
-rw-r--r-- 1 root root 13729 Apr 16 23:19 values.yaml
[root@node-exporter41 ~]# helm upgrade --install mywebui kubernetes-dashboard --create-namespace --namespace kubernetes-dashboard
Release "mywebui" does not exist. Installing it now.
NAME: mywebui
LAST DEPLOYED: Wed Apr 23 21:54:09 2025
NAMESPACE: kubernetes-dashboard
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
*************************************************************************************************
*** PLEASE BE PATIENT: Kubernetes Dashboard may need a few minutes to get up and become ready ***
*************************************************************************************************
Congratulations! You have just installed Kubernetes Dashboard in your cluster.
To access Dashboard run:
kubectl -n kubernetes-dashboard port-forward svc/mywebui-kong-proxy 8443:443
NOTE: In case port-forward command does not work, make sure that kong service name is correct.
Check the services in Kubernetes Dashboard namespace using:
kubectl -n kubernetes-dashboard get svc
Dashboard will be available at:
https://localhost:8443
3.查看部署信息
[root@node-exporter41 ~]# helm -n kubernetes-dashboard list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
mywebui kubernetes-dashboard 1 2025-04-23 21:54:09.158789016 +0800 CST deployed kubernetes-dashboard-7.12.0
[root@node-exporter41 ~]# kubectl -n kubernetes-dashboard get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mywebui-kong-6dd8f649b4-2gxqw 1/1 Running 0 72s 10.100.1.4 node-exporter42
mywebui-kubernetes-dashboard-api-5445d676c7-ggrm8 1/1 Running 0 72s 10.100.1.5 node-exporter42
mywebui-kubernetes-dashboard-auth-8687596fbd-xs8w9 1/1 Running 0 72s 10.100.0.4 node-exporter41
mywebui-kubernetes-dashboard-metrics-scraper-6ccc47c6ff-cwnq4 1/1 Running 0 72s 10.100.3.5 node-exporter43
mywebui-kubernetes-dashboard-web-584df9444c-n9sgj 1/1 Running 0 72s 10.100.0.5 node-exporter41
[root@node-exporter41 ~]#
4.修改svc的类型
[root@node-exporter41 ~]# kubectl edit svc -n kubernetes-dashboard mywebui-kong-proxy
type: NodePort
service/mywebui-kong-proxy edited
[root@node-exporter41 ~]# kubectl -n kubernetes-dashboard get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
mywebui-kong-6dd8f649b4-2gxqw 1/1 Running 0 3m18s 10.100.1.4 node-exporter42
mywebui-kubernetes-dashboard-api-5445d676c7-ggrm8 1/1 Running 0 3m18s 10.100.1.5 node-exporter42
mywebui-kubernetes-dashboard-auth-8687596fbd-xs8w9 1/1 Running 0 3m18s 10.100.0.4 node-exporter41
mywebui-kubernetes-dashboard-metrics-scraper-6ccc47c6ff-cwnq4 1/1 Running 0 3m18s 10.100.3.5 node-exporter43
mywebui-kubernetes-dashboard-web-584df9444c-n9sgj 1/1 Running 0 3m18s 10.100.0.5 node-exporter41
[root@node-exporter41 ~]# kubectl get svc -n kubernetes-dashboard mywebui-kong-proxy
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
mywebui-kong-proxy NodePort 10.200.163.82 443:7118/TCP 3m27s
5.访问WebUI
https://10.0.0.43:7118/#/login
6.创建登录账号
6.1 创建sa
[root@node-exporter41 ~]# kubectl create serviceaccount admin
serviceaccount/admin created
[root@node-exporter41 ~]#
6.2 将sa和CLuster-admin进行绑定
[root@node-exporter41 ~]# kubectl create clusterrolebinding dashboard-admin --serviceaccount=default:admin --clusterrole=cluster-admin
clusterrolebinding.rbac.authorization.k8s.io/dashboard-admin created
[root@node-exporter41 ~]#
[root@node-exporter41 ~]# kubectl create token admin
eyJhbGciOiJSUzI1NiIsImtpZCI6IlJEV1R3UmY2Wm9JSmJrdF96amVBdFZQeUZyM2p5Z1BPY1VnVnJjWE9tR2cifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLm9sZGJveWVkdS5jb20iXSwiZXhwIjoxNzQ1NDIwNDE4LCJpYXQiOjE3NDU0MTY4MTgsImlzcyI6Imh0dHBzOi8va3ViZXJuZXRlcy5kZWZhdWx0LnN2Yy5vbGRib3llZHUuY29tIiwianRpIjoiYjk5MzViMmQtNzI5YS00YmIwLThlNTEtYzI2OWUzMmMxNThlIiwia3ViZXJuZXRlcy5pbyI6eyJuYW1lc3BhY2UiOiJkZWZhdWx0Iiwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImFkbWluIiwidWlkIjoiNDFkN2I5OWEtYzFlMC00ZGEwLWExZjMtZDg0OWU1NDFiZjlhIn19LCJuYmYiOjE3NDU0MTY4MTgsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDpkZWZhdWx0OmFkbWluIn0.kcYS6zxk7bEoqwzQvIWasJgMNJRRZyiECb00VV1ty1oTsj6fk5jNCc2tQkSaHoEFp6WvTcP9-Qc99C00RNCLFFnmLiTrRkW9zGP8YfmevJUCdm3wbJ1qWyimdEcCCQcllZTMUMtvYr9gPk2kS6kCijokFv4sXWL8VsMUYg32gEIaz_o6KOGfR9BwfhQyzkQraIrp5M2-268kHwDTMdp-73C85IK7Uc4OuP93qZFCW961RHkEZY6IeXHbKwZx215J9PNxHdWxo8tWZhqV8YUX4ggeXEJUhDUlrApEJMJyC_IVQq9mn3dKv3p6Odwo6BNWKLVFR1GeswTZXic5OVUzZw
六、总结
通过以上步骤，成功部署了一个基于 Ubuntu 22.04 的 K8S 1.32 高可用集群。整个过程涵盖了从基础环境准备到 K8S 集群的完整部署，包括 Etcd 高可用集群的搭建、K8S 组件的部署、证书的生成与配置、高可用组件的安装与验证等关键环节。
这一部署过程详细展示了如何构建一个多 master 的高可用 K8S 集群，确保了在单节点故障的情况下，集群仍能正常运行，提高了系统的可靠性和稳定性。可用于企业级应用的部署，为业务的连续性和高可用性提供了有力保障。
通过精心设计的网络架构和组件配置，K8S 集群能够在节点故障时自动进行负载转移，保持服务的不间断运行。同时，采用的证书管理方案确保了集群内部通信的安全性，为后续的应用部署和扩展奠定了坚实的基础。
这一实践不仅验证了 K8S 高可用架构的可行性，还积累了宝贵的部署经验，为后续复杂应用在 K8S 上的部署和管理提供了有力支持。未来可以在这一基础上进一步扩展，如部署 Ingress 控制器、网络存储插件等，以满足更多样化的企业应用需求。