k8s部署rabbitmq集群

1 部署集群

1.1 安装

# 创建一个中间件的命名空间
kubectl create namespace middleware
# 创建ConfigMap,包含RabbitMQ的配置文件内容
kubectl apply -f rabbitmq-configmap.yaml
# 配置用于存储RabbitMQ数据的PersistentVolume（PV）和PersistentVolumeClaim（PVC）。这可以确保RabbitMQ集群的数据在节点故障时不会丢失。
kubectl apply -f rabbitmq-persistentvolume.yaml  
kubectl apply -f rabbitmq-persistentvolumeclaim.yaml
# 日志的卷
kubectl apply -f rabbitmq-log-pvc.yml
#
kubectl apply -f rabbitmq-headless.yml
# StatefulSet可以确保每个实例都有唯一的标识和稳定的网络标识符
kubectl apply -f rabbitmq-statefulset.yaml
# 创建service
kubectl apply -f rabbitmq-service.yaml

1.2 rabbitmq-configmap

apiVersion: v1
kind: ConfigMap
metadata:
  name: rabbitmq-config
  namespace: middleware
data:
  enabled_plugins: |
    [rabbitmq_management,rabbitmq_peer_discovery_k8s,rabbitmq_delayed_message_exchange].
  rabbitmq.conf: |
    cluster_formation.peer_discovery_backend  = rabbit_peer_discovery_k8s
    cluster_formation.k8s.host = kubernetes.default.svc.cluster.local
    cluster_formation.k8s.address_type = hostname
    cluster_formation.node_cleanup.interval = 10
    cluster_formation.node_cleanup.only_log_warning = true
    cluster_partition_handling = autoheal
    queue_master_locator=min-masters
    loopback_users.guest = false
    cluster_formation.randomized_startup_delay_range.min = 0
    cluster_formation.randomized_startup_delay_range.max = 2
    # 必须设置service_name，否则Pod无法正常启动，这里设置后可以不设置statefulset下env中的K8S_SERVICE_NAME变量
    cluster_formation.k8s.service_name = rabbitmq-headless
    # 必须设置hostname_suffix，否则节点不能成为集群
    cluster_formation.k8s.hostname_suffix = .rabbitmq-headless.middleware.svc.cluster.local
    # 内存上限
    vm_memory_high_watermark.absolute = 1.6GB
    # 硬盘上限
    disk_free_limit.absolute = 2G

        有的版本不识别1.6GB，故可以调整为

    # 内存上限
    vm_memory_high_watermark.absolute = 1677721600
    # 硬盘上限
    disk_free_limit.absolute = 2147483648

        执行命令，可以查看插件配置是否生效。

kubectl exec -it rabbitmq-0 -n middleware -- /bin/sh
rabbitmq-plugins list

1.3 rabbitmq-headless

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-headless
  namespace: middleware
spec:
  clusterIP: None
  ports:
  - name: amqp
    port: 5672
  - name: management
    port: 15672
  - name: cluster
    port: 25672
  selector:
    app: rabbitmq

1.4 rabbitmq-log-pvc

这里的managed-nfs-storage沿用了nacos集群的部署。

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rabbitmq-log-pvc
  namespace: middleware
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: managed-nfs-storage

        升级k8s再部署上面的log不一定能成功，可以调整为

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbitmq-log-pv
  annotations:
    pv.kubernetes.io/provisioned-by: fuseim.pri/ifs
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.101.12.95
    path: /data/k8s_storage/mid/rabbitmq/logs
  persistentVolumeReclaimPolicy: Retain
  storageClassName: managed-nfs-storage 
---
apiVersion: v1  
kind: PersistentVolumeClaim  
metadata:  
  name: rabbitmq-log-pvc  
  namespace: middleware  
spec:  
  accessModes:  
    - ReadWriteMany  
  resources:  
    requests:  
      storage: 2Gi  
  storageClassName: managed-nfs-storage

1.5 rabbitmq-persistentvolume

apiVersion: v1
kind: PersistentVolume
metadata:
  name: rabbitmq-pv
  annotations:
    pv.kubernetes.io/provisioned-by: fuseim.pri/ifs
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 10.101.12.95
    path: /data/k8s_storage/rabbitmq 
  persistentVolumeReclaimPolicy: Retain
  storageClassName: managed-nfs-storage 

1.6 rabbitmq-persistentvolumeclaim

# PersistentVolumeClaim示例  
apiVersion: v1  
kind: PersistentVolumeClaim  
metadata:  
  name: rabbitmq-pvc  
  namespace: middleware  
spec:  
  accessModes:  
    - ReadWriteMany  
  resources:  
    requests:  
      storage: 10Gi  
  storageClassName: managed-nfs-storage

1.7 rabbitmq-service

apiVersion: v1
kind: Service
metadata:
  name: rabbitmq-web-service
  namespace: middleware
spec:
  type: NodePort
  ports:
  - name: http
    port: 15672
    targetPort: 15672
    nodePort: 31567
  selector:
    app: rabbitmq

1.8 rabbitmq-statefulset

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rabbitmq
  namespace: middleware
spec:
  replicas: 3
  serviceName: rabbitmq-headless
  selector:
    matchLabels:
      app: rabbitmq
  template:
    metadata:
      labels:
        app: rabbitmq
    spec:
      serviceAccountName: rabbitmq
      containers:
      - name: rabbitmq
        image: rabbitmq:3.8.17-management
        ports:
        - containerPort: 5672
        - containerPort: 15672
        volumeMounts:
        - name: data
          mountPath: /var/lib/rabbitmq
        - name: conf
          mountPath: /etc/rabbitmq
        - name: log
          mountPath: /var/log/rabbitmq
        env:
        - name: RABBITMQ_NODENAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: RABBITMQ_ERLANG_COOKIE
          value: "nrB35tmJx5sfE3EaaYTMW7S8DxpajMrP"
        - name: RABBITMQ_LOG_BASE
          value: "/var/log/rabbitmq"
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: rabbitmq-pvc
      - name: conf
        configMap:
          name: rabbitmq-config
      - name: log
        persistentVolumeClaim:
          c