【k8s安装redis】k8s环境无pvc的情况下安装redis哨兵集群

简介

k8s集群中搭建有状态的服务会相对较麻烦，像搭建redis目前比较主流的做法主要是采用共享存储ceph、nas来实现数据的持久化存储，有的是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将redis集群的搭建采用hostpath+亲和性的权重+多副本来实现redis服务的数据持久化和高可用。

一.条件及环境说明：

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator，目前redis-operator基本都是采用了共享存储。

二.需求说明：

• 搭建redis哨兵集群。
• 高可用：有一个k8s节点死掉或重启pod之后也不影响使用。
• 高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
• 数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
• 安装简单、管理维护容易

三.实现原理及说明

• 1. 搭建redis哨兵集群，一主两从，三个哨兵进程。
• 2. redis采用三个独立的statefulset，一个副本进行搭建，哨兵进程则采用一个deployment，三个副本搭建。
• 3. redis进程采用节点亲和性及权重实现主备节点。
• 4. redis采用 service headless服务，哨兵采用service 服务。
• 5. 一组两从并根据权重主备，需要规划好六个节点的标签名称。

注：该方案搭建的redis，只有k8s集群内才能使用，不能通过LoadBlancer或NodePort的形式提供给集群外使用，因为哨兵内选举的主redis的IP都是pod的IP，当发生切换后，无法确定主redis是哪个。

四.详细步骤

4.1.规划节点标签

节点名称	角色	标签
k8s-worker-120-81	主	storage-selector=node-a-master
k8s-worker-120-82	主	storage-selector=node-b-master
k8s-worker-120-83	主	storage-selector=node-c-master
k8s-worker-120-84	备	storage-selector=node-a-slve
k8s-worker-120-85	备	storage-selector=node-a-slve
k8s-worker-120-86	备	storage-selector=node-a-slve

注：这里的角色划分是指：redis由于是一主两从，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就切换到备节点，例如：名称为defaultapp-redis-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建configmap配置

配置中包含四个文件分布说明如下：
sentinel.conf：哨兵配置文件,默认先将第一个statefulset作为主redis，其中配置：resolve-hostnames 支持主机名。
redis-master.conf：主redis配置文件，内存配置成256M，配置端口为6379，配置密码：redis#123,存储路径是/data/redis。
redis.conf：从redis配置文件,内存配置成256M，配置端口为6380，配置密码：redis#123,存储路径是/data/redis-2，并配置从主redis同步数据。
run.sh：主要作用就是判断角色运行不同的配置文件，并根据主机名创建目录，然后软连接到存储目录。这一步主要是将各自服务的redis数据存放到自己的目录，当创建新的redis的时候不会导致节点上的目录冲突。

apiVersion: v1
kind: ConfigMap
metadata:
  name: defaultapp-redis-config
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-config
data:
  sentinel.conf: |
    sentinel monitor mymaster defaultapp-redis-a-0.defaultapp-redis-a.default-ns.svc.cluster.local 6379 2
    sentinel auth-pass mymaster redis#123
    sentinel down-after-milliseconds mymaster 30000
    sentinel parallel-syncs mymaster 1
    sentinel failover-timeout mymaster 180000
    SENTINEL resolve-hostnames yes
    SENTINEL announce-hostnames yes
  redis-master.conf: |
    maxmemory 256mb
    requirepass redis#123
    save 3600 1 300 10 60 100
    dir /data/redis
    masterauth redis#123
  redis.conf: |
    maxmemory 256mb
    requirepass redis#123
    save 3600 1 300 10 60 100
    dir /data/redis
    replicaof defaultapp-redis-a-0.defaultapp-redis-a.default-ns.svc.cluster.local 6379
    masterauth redis#123
  run.sh: |
    #!/bin/sh
    role=$1
    hname=$(hostname)
    if [ $role == "master" ];then
        echo "run redis master"
        [ -d /home/redis/$hname ] || mkdir -p /home/redis/$hname
        [ -d /data ] || mkdir /data
        ln -s /home/redis/$hname /data/redis
        redis-server /etc/redis/redis-master.conf
    elif [ $role == "sentinel" ];then
        echo "run redis sentinel"
        mkdir -p /etc/redis
        cp /home/redis/sentinel.conf /etc/redis/
        redis-sentinel /etc/redis/sentinel.conf
    else
        echo "run redis slave"
        [ -d /home/redis/$hname ] || mkdir -p /home/redis/$hname
        [ -d /data ] || mkdir /data
        ln -s /home/redis/$hname /data/redis
        redis-server /etc/redis/redis.conf
    fi

4.3.创建三个statefulset和service headless配置

【1】 defaultapp-redis-a的statefulset及service headless配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: defaultapp-redis-a
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-a
spec:
  serviceName: "defaultapp-redis-a"
  replicas: 1
  selector:
    matchLabels:
      app: defaultapp-redis-a
  template:
    metadata:
      labels:
        app: defaultapp-redis-a
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-a-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-a-slave
      containers:
      - name: redis-server
        image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
        command: ["/bin/sh","/etc/redis/run.sh"]
        args: ["master"]
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/
        - name: redis-data
          mountPath: /home/redis
      restartPolicy: Always
      volumes:
        - name: redis-data
          hostPath:
            path: /data/redis_data
            type: ""
        - name: redis-config-volume
          configMap:
            name: defaultapp-redis-config
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: defaultapp-redis-a
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-a
spec:
  ports:
  - port: 6379
  clusterIP: None
  selector:
    app: defaultapp-redis-a

配置说明

      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-a-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-a-slave

该段配置主要是配置节点的亲和性和权重,当pod匹配的标签storage-selector=node-a-master 时，权重是80，而storage-selector=node-a-slave的权重只有20，则将pod分配到node-a-master节点。而当该node-a-master节点死机以后，pod产生调度会进行标签匹配，当无法匹配到node-a-master时，则会匹配到node-a-slave，这是会将pod调度到node-a-slave上。

      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600

默认情况下节点死机以后，pod根据默认值大概会在四分钟之内进行调度。该段配置是控制pod在多久时间进行调度，当节点出现not-ready和unreachable时，在3600s以内不会进行调度执行。其实该值是避免正常的维护重启或偶尔出现的网络波动导致pod频繁调度的情况。由于redis的哨兵集群是采用了三个副本，及时一个节点挂了，不影响服务。当节点长时间不服务时，可以给pod的调度设置一个合适的时间。

【2】 defaultapp-redis-b的statefulset及service headless配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: defaultapp-redis-b
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-b
spec:
  serviceName: "defaultapp-redis-b"
  replicas: 1
  selector:
    matchLabels:
      app: defaultapp-redis-b
  template:
    metadata:
      labels:
        app: defaultapp-redis-b
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-b-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-b-slave
      containers:
      - name: redis-server
        image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
        command: ["/bin/sh","/etc/redis/run.sh"]
        args: ["slave"]
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/
        - name: redis-data
          mountPath: /home/redis
      restartPolicy: Always
      volumes:
        - name: redis-data
          hostPath:
            path: /data/redis_data
            type: ""
        - name: redis-config-volume
          configMap:
            name: defaultapp-redis-config
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: defaultapp-redis-b
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-b
spec:
  ports:
  - port: 6379
  clusterIP: None
  selector:
    app: defaultapp-redis-b

【3】 defaultapp-redis-c的statefulset及service headless配置

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: defaultapp-redis-c
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-c
spec:
  serviceName: "defaultapp-redis-c"
  replicas: 1
  selector:
    matchLabels:
      app: defaultapp-redis-c
  template:
    metadata:
      labels:
        app: defaultapp-redis-c
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 80
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-c-master
          - weight: 20
            preference:
              matchExpressions:
              - key: storage-selector
                operator: In
                values:
                - node-c-slave
      containers:
      - name: redis-server
        image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 6379
        command: ["/bin/sh","/etc/redis/run.sh"]
        args: ["slave"]
        volumeMounts:
        - name: redis-config-volume
          mountPath: /etc/redis/
        - name: redis-data
          mountPath: /home/redis
      restartPolicy: Always
      volumes:
        - name: redis-data
          hostPath:
            path: /data/redis_data
            type: ""
        - name: redis-config-volume
          configMap:
            name: defaultapp-redis-config
      tolerations:
      - key: node.kubernetes.io/not-ready
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
      - key: node.kubernetes.io/unreachable
        operator: Exists
        effect: NoExecute
        tolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:
  name: defaultapp-redis-c
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-c
spec:
  ports:
  - port: 6379
  clusterIP: None
  selector:
    app: defaultapp-redis-c

4.4.创建哨兵deployment配置和service配置

哨兵进程配置文件

apiVersion: apps/v1
kind: Deployment
metadata:
  name: defaultapp-redis-sentinel
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-sentinel
spec:
  replicas: 3
  selector:
    matchLabels:
      app: defaultapp-redis-sentinel
  template:
    metadata:
      labels:
        app: defaultapp-redis-sentinel
    spec:
      containers:
      - name: redis-sentinel
        image: pcgroup-registry-vpc.cn-shenzhen.cr.aliyuncs.com/public/redis:7.2.5-alpine3.20
        ports:
        - containerPort: 26379
        command: ["/bin/sh","/home/redis/run.sh"]
        args: ["sentinel"]
        volumeMounts:
        - name: redis-config-volume
          mountPath: /home/redis
      restartPolicy: Always
      volumes:
      - name: redis-config-volume
        configMap:
          name: defaultapp-redis-config
---
apiVersion: v1
kind: Service
metadata:
  name: defaultapp-redis-sentinel
  namespace: default-ns
  labels:
    appname: default-app
    app: defaultapp-redis-sentinel
spec:
  ports:
  - port: 26379
    targetPort: 26379
  type: LoadBalancer
  selector:
    app: defaultapp-redis-sentinel

五.安装说明

• 需要先在k8s的节点创建/data/redis_data的路径来存放数据。
• 使用配置前将配置中的defaultapp替换成自己需要的命名,配置的命名空间为：default-ns，修改成自己的命名空间，appname为default-app，修改成自己的应用名称，该标签也可以删除。
• 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的redis:7.2.5-alpine3.20镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用redis:7.2.5-alpine3.20。否则配置一个可以通的镜像地址。
• redis.conf或redis-master.conf，sentinel.conf中的密码redis#123替换成自己的密码。
• 启动时是以第一个statefulset：defaultapp-redis-a 作为主节点的。
• 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。