prometheus-operator使用(五) -- 自定义pod/service自动发现配置

目标：
用户启动的service或pod，在annotation中添加label后，可以自动被prometheus发现：

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "9121"

1. secret保存自动发现的配置

若要特定的annotation被发现，需要为prometheus增加如下配置：

- job_name: 'kubernetes-service-endpoints'
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
    action: replace
    target_label: __scheme__
    regex: (https?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
    action: replace
    target_label: __metrics_path__
    regex: (.+)
  - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
    action: replace
    target_label: __address__
    regex: ([^:]+)(?::\d+)?;(\d+)
    replacement: $1:$2
  - action: labelmap
    regex: __meta_kubernetes_service_label_(.+)
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_name

上述配置会筛选endpoints：prometheus.io/scrape=True

将上述配置保存为secret:

$ kubectl create secret generic additional-configs --from-file=prometheus-additional.yaml -n monitoring
secret "additional-configs" created

2. 将配置添加到prometheus实例

修改prometheus CRD，将上面的secret添加进去：

# vi /etc/kubernetes/prometheus/prometheus-prometheus.yaml

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  labels:
    prometheus: k8s
  name: k8s
  namespace: monitoring
spec:
  ......
  additionalScrapeConfigs:
    name: additional-configs
    key: prometheus-additional.yaml
  serviceAccountName: prometheus-k8s
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  version: v2.5.0

# kubectl apply -f prometheus-prometheus.yaml

prometheus CRD修改完毕，可以到prometheus dashboard查看config是否被修改。

3. prometheus实例增加clusterrole

添加了上述配置后，prometheus-k8s-0的log会发现很多的forbidden，这是因为其没有service/pod的list权限。老的权限：

# cat /etc/kubernetes/prometheus/prometheus-clusterRole.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

需要修改其clusterRole，增加权限，新的权限：

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

执行：kubectl apply -f prometheus-clusterRole.yaml进行更新。

参考：
1.Prometheus Operator高级配置：https://www.qikqiak.com/post/...