Kubernetes的无状态应用
文章目录
- 环境
- 基础
- 示例：部署PHP和Redis的留言板应用
-
- 启动Redis数据库
-
- 创建Redis deployment
- 创建Redis service
- 设置Redis follower
- 创建Redis follower service
- 设置并暴露留言板前端
-
- 创建留言板前端deployment
- 创建前端service
- 通过 kubectl port-forward 查看前端service
- 通过负载均衡器查看前端service
- 伸缩web前端
- 清理
- 参考
环境
- RHEL 9.3
- Docker Community 24.0.7
- minikube v1.32.0
基础
创建文件 load-balancer-example.yaml 如下：
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app.kubernetes.io/name: load-balancer-example
name: hello-world
spec:
replicas: 5
selector:
matchLabels:
app.kubernetes.io/name: load-balancer-example
template:
metadata:
labels:
app.kubernetes.io/name: load-balancer-example
spec:
containers:
# - image: gcr.io/google-samples/node-hello:1.0
- image: docker.io/kaiding1/node-hello:1.0
name: hello-world
ports:
- containerPort: 8080
注：因为访问不了 gcr.io ，所以事先把image pull下来，并push到了可访问的位置。
kubectl apply -f load-balancer-example.yaml
$ kubectl get deployments hello-world
NAME READY UP-TO-DATE AVAILABLE AGE
hello-world 5/5 5 5 62s
$ kubectl describe deployment hello-world
Name: hello-world
Namespace: default
CreationTimestamp: Sat, 27 Jan 2024 08:40:21 +0800
Labels: app.kubernetes.io/name=load-balancer-example
Annotations: deployment.kubernetes.io/revision: 1
Selector: app.kubernetes.io/name=load-balancer-example
Replicas: 5 desired | 5 updated | 5 total | 5 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app.kubernetes.io/name=load-balancer-example
Containers:
hello-world:
Image: docker.io/kaiding1/node-hello:1.0
Port: 8080/TCP
Host Port: 0/TCP
Environment:
Mounts:
Volumes:
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets:
NewReplicaSet: hello-world-5756f559d7 (5/5 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 104s deployment-controller Scaled up replica set hello-world-5756f559d7 to 5
$ kubectl get replicasets
NAME DESIRED CURRENT READY AGE
hello-world-5756f559d7 5 5 5 2m5s
$ kubectl describe replicaset hello-world-5756f559d7
Name: hello-world-5756f559d7
Namespace: default
Selector: app.kubernetes.io/name=load-balancer-example,pod-template-hash=5756f559d7
Labels: app.kubernetes.io/name=load-balancer-example
pod-template-hash=5756f559d7
Annotations: deployment.kubernetes.io/desired-replicas: 5
deployment.kubernetes.io/max-replicas: 7
deployment.kubernetes.io/revision: 1
Controlled By: Deployment/hello-world
Replicas: 5 current / 5 desired
Pods Status: 5 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
Labels: app.kubernetes.io/name=load-balancer-example
pod-template-hash=5756f559d7
Containers:
hello-world:
Image: docker.io/kaiding1/node-hello:1.0
Port: 8080/TCP
Host Port: 0/TCP
Environment:
Mounts:
Volumes:
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 2m38s replicaset-controller Created pod: hello-world-5756f559d7-sstpb
Normal SuccessfulCreate 2m38s replicaset-controller Created pod: hello-world-5756f559d7-p7fjb
Normal SuccessfulCreate 2m38s replicaset-controller Created pod: hello-world-5756f559d7-r6fdd
Normal SuccessfulCreate 2m38s replicaset-controller Created pod: hello-world-5756f559d7-kkq52
Normal SuccessfulCreate 2m38s replicaset-controller Created pod: hello-world-5756f559d7-glpwp
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
hello-world-5756f559d7-glpwp 1/1 Running 0 3m26s
hello-world-5756f559d7-kkq52 1/1 Running 0 3m26s
hello-world-5756f559d7-p7fjb 1/1 Running 0 3m26s
hello-world-5756f559d7-r6fdd 1/1 Running 0 3m26s
hello-world-5756f559d7-sstpb 1/1 Running 0 3m26s
kubectl expose deployment hello-world --type=LoadBalancer --name=my-service
$ kubectl get services my-service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
my-service LoadBalancer 10.103.62.42 8080:30135/TCP 37s
注：本例中 EXTERNAL-IP 显示为
$ kubectl describe services my-service
Name: my-service
Namespace: default
Labels: app.kubernetes.io/name=load-balancer-example
Annotations:
Selector: app.kubernetes.io/name=load-balancer-example
Type: LoadBalancer
IP Family Policy: SingleStack
IP Families: IPv4
IP: 10.103.62.42
IPs: 10.103.62.42
Port: 8080/TCP
TargetPort: 8080/TCP
NodePort: 30135/TCP
Endpoints: 10.244.0.126:8080,10.244.0.127:8080,10.244.0.128:8080 + 2 more...
Session Affinity: None
External Traffic Policy: Cluster
Events:
Port:：8080/TCP NodePort:：30135/TCP Endpoints: 10.244.0.126:8080,10.244.0.127:8080,10.244.0.128:8080 + 2 more...：这些是pod的内部IP地址和端口，和下面的结果是一致的。
$ kubectl get pods --output=wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
hello-world-5756f559d7-glpwp 1/1 Running 0 14m 10.244.0.129 minikube
hello-world-5756f559d7-kkq52 1/1 Running 0 14m 10.244.0.128 minikube
hello-world-5756f559d7-p7fjb 1/1 Running 0 14m 10.244.0.127 minikube
hello-world-5756f559d7-r6fdd 1/1 Running 0 14m 10.244.0.130 minikube
hello-world-5756f559d7-sstpb 1/1 Running 0 14m 10.244.0.126 minikube
如果有外部IP地址，则可用如下命令访问应用：
curl http://-ip>:
如果是minikube环境，则使用：
minikube service：自动打开浏览器，访问应用minikube service：只列出URL，不打开浏览器访问应用--url
$ minikube service my-service --url
http://192.168.49.2:30135
注： 192.168.49.2 是node的IP地址，minikube只有一个node。
$ kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
minikube Ready control-plane 7d17h v1.28.3 192.168.49.2 Ubuntu 22.04.3 LTS 5.14.0-362.13.1.el9_3.x86_64 docker://24.0.7
$ kubectl cluster-info
Kubernetes control plane is running at https://192.168.49.2:8443
CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
效果如下：
实验完毕，删除deployment和service：
kubectl delete services my-service
kubectl delete deployment hello-world
示例：部署PHP和Redis的留言板应用
启动Redis数据库
该留言板应用使用Redis来存储数据。
创建Redis deployment
创建文件 redis-leader-deployment.yaml 如下：
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-leader
labels:
app: redis
role: leader
tier: backend
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
role: leader
tier: backend
spec:
containers:
- name: leader
image: "docker.io/redis:6.0.5"
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
kubectl apply -f redis-leader-deployment.yaml
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
redis-leader-6cc46676d8-r298l 1/1 Running 0 25s
$ kubectl logs -f deployment/redis-leader
1:C 27 Jan 2024 01:16:36.377 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 27 Jan 2024 01:16:36.377 # Redis version=6.0.5, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 27 Jan 2024 01:16:36.377 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 27 Jan 2024 01:16:36.378 * Running mode=standalone, port=6379.
1:M 27 Jan 2024 01:16:36.378 # Server initialized
1:M 27 Jan 2024 01:16:36.378 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 27 Jan 2024 01:16:36.379 * Ready to accept connections
创建Redis service
留言板应用需要与Redis通信，向其中写数据。因此，需要使用service来转发流量到Redis pod。Service定义了访问pod的策略。
创建文件 redis-leader-service.yaml 如下：
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
apiVersion: v1
kind: Service
metadata:
name: redis-leader
labels:
app: redis
role: leader
tier: backend
spec:
ports:
- port: 6379
targetPort: 6379
selector:
app: redis
role: leader
tier: backend
kubectl apply -f redis-leader-service.yaml
$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 7d18h
redis-leader ClusterIP 10.103.194.132 6379/TCP 44s
设置Redis follower
尽管Redis leader只有一个pod，可通过添加若干Redis follower，或称为副本，以使其高可用（highly available），满足流量需求。
创建文件 redis-follower-deployment.yaml 如下：
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis-follower
labels:
app: redis
role: follower
tier: backend
spec:
replicas: 2
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
role: follower
tier: backend
spec:
containers:
- name: follower
# image: us-docker.pkg.dev/google-samples/containers/gke/gb-redis-follower:v2
image: docker.io/kaiding1/gb-redis-follower:v2
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
注：因为访问不了 us-docker.pkg.dev ，所以事先把image pull下来，并push到了可访问的位置。
kubectl apply -f redis-follower-deployment.yaml
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
redis-follower-7b6577f549-jhrl8 1/1 Running 0 113s
redis-follower-7b6577f549-r9922 1/1 Running 0 113s
redis-leader-6cc46676d8-r298l 1/1 Running 0 64m
创建Redis follower service
留言板应用需要与Redis follower通信，从其中读数据。要想Redis follower可被发现，必须设置另外一个service。
创建文件 redis-follower-service.yaml 如下：
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
apiVersion: v1
kind: Service
metadata:
name: redis-follower
labels:
app: redis
role: follower
tier: backend
spec:
ports:
# the port that this service should serve on
- port: 6379
selector:
app: redis
role: follower
tier: backend
kubectl apply -f redis-follower-service.yaml
$ kubectl get service
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 443/TCP 7d18h
redis-follower ClusterIP 10.106.11.249 6379/TCP 23s
redis-leader ClusterIP 10.103.194.132 6379/TCP 31m
设置并暴露留言板前端
现在已经启动了Redis存储，接下来要启动留言板的web服务器。与Redis follower类似，前端也是通过Kubernetes的deployment来部署的。
留言板应用使用PHP前端。它被配置为与Redis follower或leader service通信，这取决于请求是读还是写。前端暴露了一个JSON接口，并提供基于jQuery-Ajax的用户体验（UX）。
创建留言板前端deployment
创建文件 frontend-deployment.yaml 如下：
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
apiVersion: apps/v1
kind: Deployment
metadata:
name: frontend
spec:
replicas: 3
selector:
matchLabels:
app: guestbook
tier: frontend
template:
metadata:
labels:
app: guestbook
tier: frontend
spec:
containers:
- name: php-redis
# image: us-docker.pkg.dev/google-samples/containers/gke/gb-frontend:v5
image: docker.io/kaiding1/gb-frontend:v5
env:
- name: GET_HOSTS_FROM
value: "dns"
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 80
kubectl apply -f frontend-deployment.yaml
$ kubectl get pods -l app=guestbook -l tier=frontend
NAME READY STATUS RESTARTS AGE
frontend-849989566c-5mqnh 1/1 Running 0 4m12s
frontend-849989566c-x74ff 1/1 Running 0 4m12s
frontend-849989566c-zvlwc 1/1 Running 0 4m12s
创建前端service
应用的Redis service只能在Kubernetes集群内部被访问，因为缺省的服务类型是 ClusterIP 。 ClusterIP 为service的pod集合提供单一的IP地址。该IP地址只能在集群内部被访问。
要想使得访客能够访问留言板，必须将前端service配置为外部可见，这样客户端可以从Kubernetes集群之外请求服务。不过，即便是使用 ClusterIP ，Kubernetes用户也可通过 kubectl port-forward 来访问service。
注意：有些云提供商，比如Google Compute Engine或者Google Kubernetes Engine，支持外部的负载均衡器。如果你的云平台支持负载均衡器，并且你希望使用它，则需反注释 type: LoadBalancer 。
创建文件 frontend-service.yaml 如下：
# SOURCE: https://cloud.google.com/kubernetes-engine/docs/tutorials/guestbook
apiVersion: v1
kind: Service
metadata:
name: frontend
labels:
app: guestbook
tier: frontend
spec:
# if your cluster supports it, uncomment the following to automatically create
# an external load-balanced IP for the frontend service.
# type: LoadBalancer
#type: LoadBalancer
ports:
# the port that this service should serve on
- port: 80
selector:
app: guestbook
tier: frontend
kubectl apply -f frontend-service.yaml
$ kubectl get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend ClusterIP 10.111.53.54 80/TCP 23s
kubernetes ClusterIP 10.96.0.1 443/TCP 7d19h
redis-follower ClusterIP 10.106.11.249 6379/TCP 31m
redis-leader ClusterIP 10.103.194.132 6379/TCP 62m
通过 kubectl port-forward 查看前端service
运行以下命令，将本机的8080端口转发到service的80端口。
$ kubectl port-forward svc/frontend 8080:80
Forwarding from 127.0.0.1:8080 -> 80
Forwarding from [::1]:8080 -> 80
打开浏览器，访问 http://localhost:8080 ，如下：
注：可能需要加代理，否则页面比较丑。
在编辑框里留言，然后提交：
通过负载均衡器查看前端service
$ kubectl get service frontend
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
frontend ClusterIP 10.111.53.54 80/TCP 20m
本例中使用的是minikube环境，所以没有外部IP地址。如果有外部IP地址，则可通过外部IP地址来访问应用。
伸缩web前端
（注：本节写的是“scale up/down”（纵向伸缩），我觉得其实应该是“scale out/in”（横向伸缩）。）
可以根据需要来伸缩应用，因为service使用了deployment控制器。
扩展前端pod数量：
kubectl scale deployment frontend --replicas=5
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
frontend-849989566c-5mqnh 1/1 Running 0 49m
frontend-849989566c-8fq67 1/1 Running 0 4s
frontend-849989566c-x74ff 1/1 Running 0 49m
frontend-849989566c-xhskx 1/1 Running 0 4s
frontend-849989566c-zvlwc 1/1 Running 0 49m
redis-follower-7b6577f549-jhrl8 1/1 Running 0 71m
redis-follower-7b6577f549-r9922 1/1 Running 0 71m
redis-leader-6cc46676d8-r298l 1/1 Running 0 133m
缩小前端pod数量：
kubectl scale deployment frontend --replicas=2
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
frontend-849989566c-5mqnh 1/1 Running 0 50m
frontend-849989566c-x74ff 1/1 Running 0 50m
redis-follower-7b6577f549-jhrl8 1/1 Running 0 72m
redis-follower-7b6577f549-r9922 1/1 Running 0 72m
redis-leader-6cc46676d8-r298l 1/1 Running 0 135m
清理
删除deployment（连带删除运行的pod）和service。通过label可用一个命令删除多个资源。
kubectl delete deployment -l app=redis
kubectl delete service -l app=redis
kubectl delete deployment frontend
kubectl delete service frontend
deployment.apps "redis-follower" deleted
deployment.apps "redis-leader" deleted
service "redis-follower" deleted
service "redis-leader" deleted
deployment.apps "frontend" deleted
service "frontend" deleted
$ kubectl get pods
No resources found in default namespace.
参考
https://kubernetes.io/docs/tutorials/stateless-application