基于eBPF的k8s网络插件Cilium部署与流量治理浅尝
1
环境准备
基于 ebpf 的 kubernetes 的 CNI 插件 cilium 最近的关注度也越来越高，并且有配套的可观测平台 hubble，为流量治理、可视化追踪有很大帮助，本文先将k8s的网络插件改为 cilium 并将 hubble 进行部署使用，具体原理请期待下篇。
Cilium 要求 Linux kernel 版本在 4.8.0 以上，Cilium 官方建议 kernel 版本至少在 4.9.17 以上，安装centos7、kubernetes、升级centos7内核可参考之前的文章。
mac安装CentOS虚拟机
使用kubeadm快速部署K8S集群
使用rpm包快速升级centos7内核版本
之前部署的kubernetes集群cni使用的是flannel，查看集群状态如下
[root@192-168-249-10 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192-168-249-10 Ready control-plane,master 253d v1.23.3
192-168-249-12 Ready 253d v1.23.3
[root@192-168-249-10 ~]# kubectl get po -nkube-system
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-6d8c4cb4d-fwx5b 1/1 Running 17 (8m51s ago) 249d
kube-system etcd-192-168-249-10 1/1 Running 26 (8m56s ago) 253d
kube-system kube-apiserver-192-168-249-10 1/1 Running 35 (8m56s ago) 253d
kube-system kube-controller-manager-192-168-249-10 1/1 Running 22 (8m56s ago) 253d
kube-system kube-flannel-ds-4pr9g 1/1 Running 5 (41s ago) 251d
kube-system kube-flannel-ds-d86kv 1/1 Running 18 (8m56s ago) 251d
kube-system kube-proxy-hnrws 1/1 Running 18 (8m56s ago) 253d
kube-system kube-proxy-k9tkw 1/1 Running 5 (41s ago) 253d
kube-system kube-scheduler-192-168-249-10 1/1 Running 58 (8m56s ago) 253d 卸载flannel
# kubectl delete -f flannel.yml
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy "psp.flannel.unprivileged" deleted
clusterrole.rbac.authorization.k8s.io "flannel" deleted
clusterrolebinding.rbac.authorization.k8s.io "flannel" deleted
serviceaccount "flannel" deleted
configmap "kube-flannel-cfg" deleted
daemonset.apps "kube-flannel-ds" deletedmv /var/lib/cni/ /var/lib/cni.bak
mv /etc/cni/net.d/10-flannel.conflist /etc/cni/net.d/10-flannel.conflist.bak
systemctl restart kubelet卸载完 flannel，再次查看集群状态，所有节点应为 notReady 状态
# kubectl get node
NAME STATUS ROLES AGE VERSION
192-168-249-10 NotReady control-plane,master 253d v1.23.3
192-168-249-12 NotReady 253d v1.23.3 2
安装cilium
我们下载cilium命令行工具，然后通过命令行工具安装cilium。
# curl -L --remote-name-all https://github.com/cilium/cilium-cli/releases/latest/download/cilium-linux-amd64.tar.gz
# tar -zxvf cilium-linux-amd64.tar.gz
# mv cilium /usr/local/bin/
# cilium install
安装完成后，可以通过cilium工具查看cilium状态。
cilium status
查看集群状态，所有节点变成Ready状态，所有pod为Running状态。
[root@192-168-249-10 ~]# kubectl get node
NAME STATUS ROLES AGE VERSION
192-168-249-10 Ready control-plane,master 254d v1.23.3
192-168-249-12 Ready 254d v1.23.3
[root@192-168-249-10 ~]# kubectl get po -nkube-system
NAME READY STATUS RESTARTS AGE
cilium-njq7m 1/1 Running 0 2m17s
cilium-nln6m 1/1 Running 0 2m17s
cilium-operator-9dc4b59f7-czjtn 1/1 Running 0 2m17s
coredns-6d8c4cb4d-5hg6n 1/1 Running 0 91s
etcd-192-168-249-10 1/1 Running 26 (46h ago) 254d
kube-apiserver-192-168-249-10 1/1 Running 35 (46h ago) 254d
kube-controller-manager-192-168-249-10 1/1 Running 22 (46h ago) 254d
kube-proxy-hnrws 1/1 Running 18 (46h ago) 254d
kube-proxy-k9tkw 1/1 Running 5 (46h ago) 254d
kube-scheduler-192-168-249-10 1/1 Running 58 (46h ago) 254d 3
安装hubble
Hubble是建立在Cilium和eBPF之上，以一种完全透明的方式，提供网络基础设施通信以及应用行为的深度可视化，是一个应用于云原生工作负载，完全分布式的网络和安全可观察性平台。
可以直接使用 cilium 工具安装 hubble 和 hubble ui 工具。
[root@192-168-249-10 ~]# cilium hubble enable
Found CA in secret cilium-ca
ℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.12.2 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,hubble.enabled=true,hubble.relay.enabled=true
✨ Patching ConfigMap cilium-config to enable Hubble...
Creating ConfigMap for Cilium version 1.12.2...
♻️ Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
Creating Peer Service...
✨ Generating certificates...
Generating certificates for Relay...
✨ Deploying Relay...
⌛ Waiting for Hubble to be installed...
ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
✅ Hubble was successfully enabled!如果安装时报错 service 已存在，就先删除这个 svc 再安装一次。
[root@192-168-249-10 ~]# cilium hubble enable --ui
Error: Unable to enable Hubble: services "hubble-peer" already exists
[root@192-168-249-10 ~]# kubectl delete svc -nkube-system hubble-peer
[root@192-168-249-10 ~]# cilium hubble enable --ui
Found CA in secret cilium-ca
ℹ️ helm template --namespace kube-system cilium cilium/cilium --version 1.12.2 --set cluster.id=0,cluster.name=kubernetes,encryption.nodeEncryption=false,hubble.enabled=true,hubble.relay.enabled=true
✨ Patching ConfigMap cilium-config to enable Hubble...
Creating ConfigMap for Cilium version 1.12.2...
♻️ Restarted Cilium pods
⌛ Waiting for Cilium to become ready before deploying other Hubble component(s)...
Creating Peer Service...
✅ Relay is already deployed
✨ Deploying Hubble UI and Hubble UI Backend...
⌛ Waiting for Hubble to be installed...
ℹ️ Storing helm values file in kube-system/cilium-cli-helm-values Secret
✅ Hubble was successfully enabled通过 cilium status 命令查看，发现 hubble 的状态变成 ok
4
hubble 流量治理
下载 hubble 命令行工具
[root@192-168-249-10 ~]# export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
[root@192-168-249-10 ~]# echo $HUBBLE_VERSION
v0.10.0
[root@192-168-249-10 ~]# curl -L --remote-name-all https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
[root@192-168-249-10 ~]# tar -zxvf hubble-linux-amd64.tar.gz -C /usr/local/bin/使用 hubble observe 命令查看流量
[root@192-168-249-10 ~]# cilium hubble port-forward&
[1] 24941
[root@192-168-249-10 ~]# hubble status
Healthcheck (via localhost:4245): Ok
Current/Max Flows: 3,347/8,190 (40.87%)
Flows/s: 4.55
Connected Nodes: 2/2
[root@192-168-249-10 ~]# hubble observe
使用 hubble ui 工具查看容器间访问关系
[root@192-168-249-10 ~]# cilium hubble ui
ℹ️ Opening "http://localhost:12000" in your browser...
至此我们就已经部署完成 cilium 和 hubble 可观测平台，ebpf 的底层原理、cilium 的各种优势与能力，请期待下篇，欢迎关注～
点个赞
再走吧