k8s 1.29 一键安装脚本, 丝滑致极
博客原文
文章目录
-
- 集群配置
-
- 配置清单
- 集群规划
- 集群网络规划
- 环境初始化
-
- 主机配置
- 安装脚本
-
- 需要魔法的脚本
- 不需要魔法的脚本
- 配置自动补全
- 加入其余节点
- 验证集群
高可用版本: 高可用 k8s 1.29 一键安装脚本
集群配置
配置清单
- OS： ubuntu 20.04
- kubernetes： 1.29.1
- Container Runtime：Containerd 1.7.11
- CRI: runc 1.10
- CNI: cni-plugin 1.4
集群规划
| IP | Hostname | 配置 |
|---|---|---|
| 192.168.254.130 | master01 | 2C 4G 30G |
| 192.168.254.131 | node01 | 2C 4G 30G |
| 192.168.254.132 | node02 | 2C 4G 30G |
集群网络规划
- Pod 网络: 10.244.0.0/16
- Service 网络: 10.96.0.0/12
- Node 网络: 11.0.1.0/24
环境初始化
主机配置
ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]
# 将节点加入 hosts
cat << EOF >> /etc/hosts
192.168.254.130 master01
192.168.254.131 node01
192.168.254.132 node02
EOF
安装脚本
**前置条件: ** 脚本中存在拉取国外资源, 需要你配置代理 ==> [如何让虚拟机拥有愉快网络环境](https://ai-feier.github.io/p/%E5%A6%82%E4%BD%95%E8%AE%A9%E8%99%9A%E6%8B%9F%E6%9C%BA%E6%8B%A5%E6%9C%89%E6%84%89%E5%BF%AB%E7%BD%91%E7%BB%9C%E7%8E%AF%E5%A2%83/)
需要:
- 虚拟机代理
- apt 下载代理
在所有节点执行以下脚本
脚本功能:
- 时间同步
- 关闭 swap
- 启用内核模块
- 安装 ipvs 并启用内核参数
- 安装 containerd, runc, cni
- 更改 containerd 沙箱镜像和 cgroup 并且配置镜像加速
- 安装最新 kubelet, kubeadm, kubectl
注意: 请先通过export name=master01方式设置当前 node 的 hostname
需要魔法的脚本
install.sh:
export name=master01 # 改为你 hostname 的名称, 脚本中删除该行
#!/bin/bash
hostnamectl set-hostname $name
# 阿里源
mv /etc/apt/sources.list /etc/apt/sources.list.bak
cat <<EOF > /etc/apt/sources.list
deb https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
EOF
apt update
# 时间同步
timedatectl set-timezone Asia/Shanghai
#安装chrony，联网同步时间
apt install chrony -y && systemctl enable --now chronyd
# 禁用 swap
sudo swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
# 安装 ipvs
apt install -y ipset ipvsadm
# 配置需要的内核模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
# 启动模块
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 是 sysctl 参数生效
sudo sysctl --system
# 检验是否配置成功
#lsmod | grep br_netfilter
#lsmod | grep overlay
#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
# 配置 ipvs 内核参数
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF
# 内核加载 ipvs
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack
# 确认ipvs模块加载
#lsmod |grep -e ip_vs -e nf_conntrack
# 安装 Containerd
wget -c https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz
tar -xzvf containerd-1.7.11-linux-amd64.tar.gz
#解压出来一个bin目录,containerd可执行文件都在bin目录里面
mv bin/* /usr/local/bin/
rm -rf bin
#使用systemcd来管理containerd
cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now containerd
#systemctl status containerd
# 安装 runc
#runc是容器运行时，runc实现了容器的init，run，create，ps...我们在运行容器所需要的cmd：
curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64 && \
install -m 755 runc.amd64 /usr/local/sbin/runc
# 安装 CNI plugins
wget -c https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
#根据官网的安装步骤来，创建一个目录用于存放cni插件
mkdir -p /opt/cni/bin
tar -xzvf cni-plugins-linux-amd64-v1.4.0.tgz -C /opt/cni/bin/
# 修改 Containd 配置
#修改containerd的配置，因为containerd默认从k8s官网拉取镜像
#创建一个目录用于存放containerd的配置文件
mkdir -p /etc/containerd
#把containerd配置导出到文件
containerd config default | sudo tee /etc/containerd/config.toml
# 修改沙箱镜像
sed -i 's#sandbox_image = "registry.k8s.io/pause:.*"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#' /etc/containerd/config.toml
# 修改 cgroup 为 systemd
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml
# 配置镜像加速
sed -i 's#config_path = ""#config_path = "/etc/containerd/certs.d"#' /etc/containerd/config.toml
# 配置 Containerd 镜像源
# docker hub镜像加速
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://dockerproxy.com"]
capabilities = ["pull", "resolve"]
[host."https://docker.m.daocloud.io"]
capabilities = ["pull", "resolve"]
[host."https://reg-mirror.qiniu.com"]
capabilities = ["pull", "resolve"]
[host."https://registry.docker-cn.com"]
capabilities = ["pull", "resolve"]
[host."http://hub-mirror.c.163.com"]
capabilities = ["pull", "resolve"]
EOF
# k8s.gcr.io镜像加速
mkdir -p /etc/containerd/certs.d/k8s.gcr.io
tee /etc/containerd/certs.d/k8s.gcr.io/hosts.toml << 'EOF'
server = "https://k8s.gcr.io"
[host."https://k8s-gcr.m.daocloud.io"]
capabilities = ["pull", "resolve", "push"]
EOF
#重启containerd
systemctl restart containerd
#systemctl status containerd
# 安装 kubeadm、kubelet、kubectl
# 安装依赖
sudo systemctl restart containerd
sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -y
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
# kubelet 开机自启
systemctl enable --now kubelet
# 配置 crictl socket
crictl config runtime-endpoint unix:///run/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock
不需要魔法的脚本
前置:
下载我下载好的资源包
- 阿里云 OSS
- CSDN 资源
资源列表:
Containerd: - Container Runtime：Containerd 1.7.11 - CRI: runc 1.10 - CNI: cni-plugin 1.4 calico 3.27: - tigera-operator.yaml - custom-resources.yaml
| 资源 | 原始地址 |
|---|---|
| Container Runtime：Containerd 1.7.11 | https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz |
| CRI: runc 1.10 | https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64 |
| CNI: cni-plugin 1.4 | https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz |
| calico 3.27 : tigera-operator.yaml | https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/tigera-operator.yaml |
| calico 3.27 : custom-resources.yaml | https://raw.githubusercontent.com/projectcalico/calico/v3.27.0/manifests/custom-resources.yaml |
下载资源:
wget -O - https://blog-source-mkt.oss-cn-chengdu.aliyuncs.com/resources/k8s/kubeadm%20init/k8s1.29.tar.gz | tar xzvf -
cd workdir
export name=master01 # 改为你 hostname 的名称
install.sh
#!/bin/bash
hostnamectl set-hostname $name
# 阿里源
mv /etc/apt/sources.list /etc/apt/sources.list.bak
cat <<EOF > /etc/apt/sources.list
deb https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiversedeb https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
EOF
apt update
# 时间同步
timedatectl set-timezone Asia/Shanghai
#安装chrony，联网同步时间
apt install chrony -y && systemctl enable --now chronyd
# 禁用 swap
sudo swapoff -a && sed -i '/swap/s/^/#/' /etc/fstab
# 安装 ipvs
apt install -y ipset ipvsadm
# 配置需要的内核模块
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
# 启动模块
sudo modprobe overlay
sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward = 1
EOF
# 是 sysctl 参数生效
sudo sysctl --system
# 检验是否配置成功
#lsmod | grep br_netfilter
#lsmod | grep overlay
#sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
# 配置 ipvs 内核参数
cat <<EOF | sudo tee /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
EOF
# 内核加载 ipvs
sudo modprobe ip_vs
sudo modprobe ip_vs_rr
sudo modprobe ip_vs_wrr
sudo modprobe ip_vs_sh
sudo modprobe nf_conntrack
# 确认ipvs模块加载
#lsmod |grep -e ip_vs -e nf_conntrack
# 安装 Containerd
#wget -c https://github.com/containerd/containerd/releases/download/v1.7.11/containerd-1.7.11-linux-amd64.tar.gz
tar -xzvf containerd-1.7.11-linux-amd64.tar.gz
#解压出来一个bin目录,containerd可执行文件都在bin目录里面
mv bin/* /usr/local/bin/
rm -rf bin
#使用systemcd来管理containerd
cat << EOF > /usr/lib/systemd/system/containerd.service
[Unit]
Description=containerd container runtime
Documentation=https://containerd.io
After=network.target local-fs.target
[Service]
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/containerd
Type=notify
Delegate=yes
KillMode=process
Restart=always
RestartSec=5
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Comment TasksMax if your systemd version does not supports it.
# Only systemd 226 and above support this version.
TasksMax=infinity
OOMScoreAdjust=-999
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload && systemctl enable --now containerd
#systemctl status containerd
# 安装 runc
#runc是容器运行时，runc实现了容器的init，run，create，ps...我们在运行容器所需要的cmd：
#curl -LO https://github.com/opencontainers/runc/releases/download/v1.1.10/runc.amd64 && \
install -m 755 runc.amd64 /usr/local/sbin/runc
# 安装 CNI plugins
#wget -c https://github.com/containernetworking/plugins/releases/download/v1.4.0/cni-plugins-linux-amd64-v1.4.0.tgz
#根据官网的安装步骤来，创建一个目录用于存放cni插件
mkdir -p /opt/cni/bin
tar -xzvf cni-plugins-linux-amd64-v1.4.0.tgz -C /opt/cni/bin/
# 修改 Containd 配置
#修改containerd的配置，因为containerd默认从k8s官网拉取镜像
#创建一个目录用于存放containerd的配置文件
mkdir -p /etc/containerd
#把containerd配置导出到文件
containerd config default | sudo tee /etc/containerd/config.toml
# 修改沙箱镜像
sed -i 's#sandbox_image = "registry.k8s.io/pause:.*"#sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"#' /etc/containerd/config.toml
# 修改 cgroup 为 systemd
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#' /etc/containerd/config.toml
# 配置镜像加速
sed -i 's#config_path = ""#config_path = "/etc/containerd/certs.d"#' /etc/containerd/config.toml
# 配置 Containerd 镜像源
# docker hub镜像加速
mkdir -p /etc/containerd/certs.d/docker.io
cat > /etc/containerd/certs.d/docker.io/hosts.toml << EOF
server = "https://docker.io"
[host."https://dockerproxy.com"]
capabilities = ["pull", "resolve"]
[host."https://docker.m.daocloud.io"]
capabilities = ["pull", "resolve"]
[host."https://reg-mirror.qiniu.com"]
capabilities = ["pull", "resolve"]
[host."https://registry.docker-cn.com"]
capabilities = ["pull", "resolve"]
[host."http://hub-mirror.c.163.com"]
capabilities = ["pull", "resolve"]
EOF
# k8s.gcr.io镜像加速
mkdir -p /etc/containerd/certs.d/k8s.gcr.io
tee /etc/containerd/certs.d/k8s.gcr.io/hosts.toml << 'EOF'
server = "https://k8s.gcr.io"
[host."https://k8s-gcr.m.daocloud.io"]
capabilities = ["pull", "resolve", "push"]
EOF
#重启containerd
systemctl restart containerd
#systemctl status containerd
# 安装 kubeadm、kubelet、kubectl
# 安装依赖
sudo systemctl restart containerd
sudo apt-get update -y
sudo apt-get install -y apt-transport-https ca-certificates curl gpg
mkdir -p /etc/apt/keyrings
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.29/deb/Release.key | sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] https://pkgs.k8s.io/core:/stable:/v1.29/deb/ /' | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update -y
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl
# kubelet 开机自启
systemctl enable --now kubelet
# 配置 crictl socket
crictl config runtime-endpoint unix:///run/containerd/containerd.sock
crictl config image-endpoint unix:///run/containerd/containerd.sock
chmod +x install.sh
./install.sh
在你的主节点初始化集群(同样在 workdir/ 下)
export POD_CIDR=10.244.0.0/16
export SERVICE_CIDR=10.96.0.0/12
export APISERVER_MASTER01=192.168.254.130
kubeadm init \
--apiserver-advertise-address=$APISERVER_MASTER01 \
--apiserver-bind-port=6443 \
--image-repository registry.aliyuncs.com/google_containers \
--kubernetes-version v1.29.1 \
--service-cidr=$SERVICE_CIDR \
--pod-network-cidr=$POD_CIDR --upload-certs
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
# 安装 calico
sed -i 's#cidr.*#cidr: '$POD_CIDR'#' custom-resources.yaml
kubectl create -f tigera-operator.yaml
kubectl create -f custom-resources.yaml
此时, 你的 k8s 集群就已经搭建成功了, 但是, kube-proxy 使用的依然是 iptables, 我们需要将其手动更换为 ipvs
kubectl -n kube-system edit cm kube-proxy
找到:
mode: ""
更改为: ==>
mode: "ipvs"
k8s 集群就万事大吉了, 如果你想把已经搭建好的集群升级为高可用集群 | 传送: [keepalived+nginx实现高可用apiserver](https://ai-feier.github.io/p/keepalived-nginx%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8apiserver)
配置自动补全
apt install bash-completion -y
cat << EOF >> ~/.profile
alias k='kubectl'
source <(kubectl completion bash)
complete -F __start_kubectl k
EOF
source ~/.profile
加入其余节点
在其余节点执行
$ kubeadm join 192.168.254.130:6443 --token ub130l.4i7hcdhk9c0g5nz6 \
--discovery-token-ca-cert-hash sha256:56c8eafbd4c8c37ea88dd4690f4e7b38b5773c3b64b97a1165f5961b0450b0ac
验证集群
$ k get po -A
NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-5c7b4b46d6-hd5mv 0/1 Pending 0 2m5s
calico-system calico-node-4nkcf 0/1 Init:1/2 0 2m5s
calico-system calico-node-gkwpf 0/1 Init:1/2 0 60s
calico-system calico-node-xb222 0/1 Init:1/2 0 56s
calico-system calico-typha-599c7784cf-kkmpf 0/1 ContainerCreating 0 2m6s
calico-system calico-typha-599c7784cf-ktrdr 0/1 ContainerCreating 0 47s
calico-system csi-node-driver-2cfhm 0/2 ContainerCreating 0 60s
calico-system csi-node-driver-rwtnq 0/2 ContainerCreating 0 56s
calico-system csi-node-driver-wq486 0/2 ContainerCreating 0 2m5s
kube-system coredns-857d9ff4c9-fz2z6 0/1 Pending 0 2m26s
kube-system coredns-857d9ff4c9-j6247 0/1 Pending 0 2m26s
kube-system etcd-master01 1/1 Running 0 2m39s
kube-system kube-apiserver-master01 1/1 Running 0 2m39s
kube-system kube-controller-manager-master01 1/1 Running 0 2m41s
kube-system kube-proxy-nxtgf 1/1 Running 0 60s
kube-system kube-proxy-sps8j 1/1 Running 0 2m26s
kube-system kube-proxy-vgh2g 1/1 Running 0 56s
kube-system kube-scheduler-master01 1/1 Running 0 2m39s
tigera-operator tigera-operator-55585899bf-sfvkf 1/1 Running 0 2m26s
参考:
- https://kubernetes.io/zh-cn/docs/setup/production-environment/tools/kubeadm/install-kubeadm/
- https://ai-feier.github.io/p/keepalived-nginx%E5%AE%9E%E7%8E%B0%E9%AB%98%E5%8F%AF%E7%94%A8apiserver/
- https://blog.csdn.net/m0_51964671/article/details/135256571