Prometheus和Grafana持久化存储
之前部署的数据是在临时的存储目录里面，当pod重启或者被删除后，数据也就没了 对于Prometheus监控来说根据需求保存1周或者1个月，但是一定要持久化存储
一、配置Prometheus数据持久化
查看之前创建的nfs动态存储
[root@master01 manifests]# kubectl get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-aqsc nfs-provisioner-01 Retain Immediate false 23h查看需要存储的pod
[root@master01 manifests]# kubectl get pods -n monitoring | grep prometheus-k8s
prometheus-k8s-0 2/2 Running 1 39m
prometheus-k8s-1 2/2 Running 1 39mPrometheus用的是有状态服务，StatefulSet 用户不需要手工创建PVC 通过在Pod的spec中指定storageClassName和所需的存储资源来触发动态PVC的创建。
[root@master01 manifests]# kubectl get pods -n monitoring prometheus-k8s-1 -o yaml | grep kind
kind: Pod
f:kind: {}
kind: StatefulSet
#可以看到 使用的是有状态存储[root@master01 manifests]# kubectl get statefulsets.apps -n monitoring
NAME READY AGE
alertmanager-main 3/3 16h
prometheus-k8s 2/2 16h #可以在有状态服务里面 找到他那么只需要提需求就可以了
1 在线修改
[root@master01 manifests]# kubectl edit prometheus -n monitoring k8s
......
......
name: k8s
namespace: monitoring
resourceVersion: "10762913"
uid: 167328a1-df5a-4249-a107-fc7189f9f20c
spec:
storage: #将这些加到spec下面
volumeClaimTemplate:
spec:
accessModes: #模式
- ReadWriteOnce #只能被单个pod读写
storageClassName: "nfs-nfs-aqsc" #storageclasses名字 nfs的
resources:
requests:
storage: 1Gi #限制 这个对nfs没用保存退出
2 yaml文件修改
[root@master01 manifests]# vim prometheus-prometheus.yaml
spec:
storage: #将这些加到spec下面
volumeClaimTemplate:
spec:
accessModes: #模式
- ReadWriteOnce #只能被单个pod读写
storageClassName: "nfs-aqsc" #storageclasses名字 nfs的
resources:
requests:
storage: 1Gi #限制 这个对nfs没用然后创建了两个这样就做到持久化存储了
[root@master01 manifests]# kubectl get pv -n monitoring | grep 1Gi
pvc-2f1f4a6c-73b8-4697-a5e8-8e86b0f9cb6d 1Gi RWO Retain Bound monitoring/prometheus-k8s-db-prometheus-k8s-1 nfs-aqsc 2m12s
pvc-a1c2a276-881f-4008-9d6d-1a842622aeaa 1Gi RWO Retain Bound monitoring/prometheus-k8s-db-prometheus-k8s-0 nfs-aqsc 2m12sprometheus的两个pod重启了
[root@master01 manifests]# kubectl get pods -n monitoring | grep prometheus-k8s
prometheus-k8s-0 2/2 Running 1 2m23s
prometheus-k8s-1 2/2 Running 1 2m23s到此Prometheus的持久化存储就完成了
查看一下
[root@master01 manifests]# kubectl get pods -n monitoring prometheus-k8s-1 -o wide
NAME READY STATUS RESTARTS AGE IP NODE GATES
prometheus-k8s-1 2/2 Running 1 2m36s 10.244.15.158 10.1.19.54 #得到node的IP
#去node ip主机上查看下
[root@node01 ~]# ip a | grep 10.1.19.54
inet 10.1.19.54/24 brd 10.1.19.255 scope global noprefixroute eth0
[root@node01 ~]# df -hT | grep "prometheus"
tmpfs tmpfs 5.3G 12K 5.3G 1% /data/kubelet/pods/1fb7ba7d-cfcb-4803-af22-185d8797e70d/volumes/kubernetes.io~secret/prometheus-k8s-token-z74dp
10.1.19.57:/nfs/monitoring-prometheus-k8s-db-prometheus-k8s-1-pvc-2f1f4a6c-73b8-4697-a5e8-8e86b0f9cb6d nfs4 300G 2.2G 298G 1% /data/kubelet/pods/1fb7ba7d-cfcb-4803-af22-185d8797e70d/volumes/kubernetes.io~nfs/pvc-2f1f4a6c-73b8-4697-a5e8-8e86b0f9cb6d
10.1.19.57:/nfs/monitoring-prometheus-k8s-db-prometheus-k8s-1-pvc-2f1f4a6c-73b8-4697-a5e8-8e86b0f9cb6d/prometheus-db nfs4 300G 2.2G 298G 1% /data/kubelet/pods/1fb7ba7d-cfcb-4803-af22-185d8797e70d/volume-subpaths/pvc-2f1f4a6c-73b8-4697-a5e8-8e86b0f9cb6d/prometheus/2
#可以看到挂载信息二、配置Grafana的持久化存储
查看之前创建的nfs动态存储
[root@master01 manifests]# kubectl get storageclasses.storage.k8s.io
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
nfs-aqsc nfs-provisioner-01 Retain Immediate false 23hGrafana是无状态服务，我们使用nfs动态存储类 只需要创建一个pvc 就会自动给我们创建合适pv
[root@master01 manifests]# kubectl get deployments.apps -n monitoring
NAME READY UP-TO-DATE AVAILABLE AGE
blackbox-exporter 1/1 1 1 16h
grafana 1/1 1 1 16h #可以看到Grafana是无状态应用
kube-state-metrics 1/1 1 1 16h
prometheus-adapter 2/2 2 2 16h
prometheus-operator 1/1 1 1 17h看下现在grafana使用的存储
[root@master01 manifests]# kubectl get -n monitoring pods grafana-f6bfc8ffd-lw9sh -oyaml | grep -A5 volumes:
f:volumes:
.: {}
k:{"name":"grafana-dashboard-apiserver"}:
.: {}
f:configMap:
.: {}
--
volumes:
- emptyDir: {} #现在使用的临时存储 如果pod重启或者删除 那么存储的数据就会删除 消失
name: grafana-storage
- name: grafana-datasources
secret:
defaultMode: 420写一个pvc
[root@master01 manifests]# cat pvc-grafana.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: grafana-pvc-nfs
namespace: monitoring
spec:
accessModes: #使用的模式
- ReadWriteMany #可以被多个节点同时读写
resources:
requests:
storage: "2Gi" #存储大小为2G 不过对于NFS来说 这个没用 限制不了
storageClassName: nfs-aqsc #使用的storageclass名字############旧的配置
kind: deployment
spec
template:
sepc:
.......
......
volumes:
- emptyDir: {} #这行删掉
name: grafana-storage #这行删掉
#改为新的配置
kind: deployment
spec
template:
sepc:
......
......
volumes:
- name: grafana-storage #这行加上
persistentVolumeClaim: #这行加上
claimName: grafana-pvc-nfs #这行加上
readOnly: false #这行加上查看下挂载的信息
[root@master01 manifests]# kubectl get pods -n monitoring grafana-7857bc4b59-lgbrz -owide
NAME READY STATUS RESTARTS AGE IP NODE
grafana-7857bc4b59-lgbrz 1/1 Running 0 2m16s 10.244.171.82 10.1.19.55 #pod所在node ip
#去node ip主机上查看下
[root@node02 ~]# ip a | grep 10.1.19.55
inet 10.1.19.55/24 brd 10.1.19.255 scope global noprefixroute eth0
[root@node02 ~]# df -hT | grep grafana
tmpfs tmpfs 5.3G 12K 5.3G 1% /data/kubelet/pods/7b22798a-d828-4b9f-bca1-df79f251e090/volumes/kubernetes.io~secret/grafana-token-vb2r5
tmpfs tmpfs 5.3G 4.0K 5.3G 1% /data/kubelet/pods/7b22798a-d828-4b9f-bca1-df79f251e090/volumes/kubernetes.io~secret/grafana-datasources
10.1.19.57:/nfs/monitoring-grafana-pvc-nfs-pvc-67001dc9-bd77-4e49-a278-dc567389c8f4 nfs4 300G 2.2G 298G 1% /data/kubelet/pods/7b22798a-d828-4b9f-bca1-df79f251e090/volumes/kubernetes.io~nfs/pvc-67001dc9-bd77-4e49-a278-dc567389c8f4