kube-prometheus-stack+thanos+minio经验

一、准备
1、k8s 1.23
2、helm 3.8
3、minio最新版本 （请自行安装，本人使用docker部暑单节点）
4、kube-prometheus-stack 版本为：35.0.0 （helm安装）
5、kube-thanos版本为：10.3.6 （依赖于第4步，使用helm安装）

二、目标
kube-thanos实现安装以下组件

• query #查询 （通过prometheus和storegateway）
• compactor #去重
• storegateway #为query提供查询objstore
• sidecar #在kube-prometheus-stack安装时已安装, 用于数据上传和query查询
• ruler #（可不安装）

三、在kube-prometheus-stack已要正常运行后，开启thanos-sidecar实现数据上传到objstore(Minio)的配置

1、kube-prometheus-stack的values.yaml配置

prometheus:
  thanosService:
    enabled: true
  extraSecret:                                    #配置thanos的bucket-config, 里面包括了objstor(minio)的配置
    name: bucket-config
    data:
      objstore.yml: |
        type: S3
        config:
          bucket: "thanos-buckets01"                 #minio的桶名
          endpoint: "10.33.xxx.xxx:9000"             #minio的地址
          access_key: "Thanos"                       #minio的帐号
          secret_key: "Thanos@654321"                #minio的密码
          insecure: true                             #不验证tls证书
  prometheusSpec:
    disableCompaction: true                          #kube-prometheus-stack 启用thanos-sidecar
    thanos:
      objectStorageConfig:                           #thanos使用上边的secret来配置thanos-sidecar
        name: bucket-config
        key: objstore.yml

本图是thanos-sidecar上传数据到minio的结果

四、kube-thanos安装
1、下载charts

#指定变量
pro=thanos
chart_version=10.3.6

mkdir -p /data/$pro
cd /data/$pro

#下载charts
helm pull bitnami/$pro --version=$chart_version

#提取values.yaml文件
tar zxvf $pro-$chart_version.tgz --strip-components 1 $pro/values.yaml 

cat > /data/$pro/start.sh << EOF
kubectl get ns monitoring||kubectl create ns monitoring
helm install $pro $pro-$chart_version.tgz \
-f values.yaml \
-n monitoring
EOF

#helm3 升级
cat > /data/$pro/upgrade.sh << EOF
helm install $pro $pro-$chart_version.tgz \
-f values.yaml \
-n monitoring
cp values.yaml values.yaml.bak_`date +%F_%R`
EOF

二、 配置values.yaml

#此处对应kube-prometheus-stack的values.yaml配置中的prometheus.extraSecret.name
existingObjstoreSecret: "bucket-config"
query:
  enabled: true
  replicaLabel: [prometheus_replica]                          #去重标记
  dnsDiscovery:
    enabled: true
    sidecarsService: "kube-prometheus-stack-thanos-discovery"  #kube-prometheus-stack的thanos-servicename
    sidecarsNamespace: "monitoring"                            #kube-prometheus-stack部暑空间   
  ingress:
    enabled: true
    hostname: thanos.lady.cn    #域名，自定义就好
queryFrontend:                 #提供给grafana查询使用，看下图
  enabled: true 
compactor:
  enabled: true
  persistence:
    enabled: false          #生产环境设为true,持久化
storegateway:
  enabled: true 
  persistence:
    enabled: false           #生产环境设为true,持久化
ruler:
  enabled: true
  replicaLabel: prometheus_replica              #去重标记
  alertmanagers:
  - kube-prometheus-stack-alertmanager:9093       #kube-prometheus-stack的servicename地址
  existingConfigmap: "kube-prometheus-stack-alertmanager-overview"   ##kube-prometheus-stack的alertmanagers配置
  persistence:
    enabled: false             #生产环境设为true,持久化
  ingress:
    enabled: true
    hostname: thanos-ruler.lady.cn     #域名，自定义就好

query图，包含了sidecar、store、rule

• grafana配置新的数据源为 http://thanos-query-forntend:9090/
  
  

[root@host-192-168-11-100 kube-thanos]# kubectl get pod -A
NAMESPACE     NAME                                                        READY   STATUS    RESTARTS   AGE
kube-system   calico-kube-controllers-6d948fc787-gpsw2                    1/1     Running   0          2d
kube-system   calico-node-9h4nq                                           1/1     Running   0          2d
kube-system   coredns-c79c957c8-96jf5                                     1/1     Running   0          2d
kube-system   dashboard-metrics-scraper-8464848978-86gqb                  1/1     Running   0          2d
kube-system   kubernetes-dashboard-9457cbb47-7hq2r                        1/1     Running   0          2d
kube-system   metrics-server-6d6786c9db-clstn                             1/1     Running   0          2d
kube-system   traefik2-74f8d7b659-z5ldb                                   1/1     Running   0          4h37m
monitoring    alertmanager-kube-prometheus-stack-alertmanager-0           2/2     Running   0          30h
monitoring    kube-prometheus-stack-grafana-799446c5b9-q5hhh              3/3     Running   0          30h
monitoring    kube-prometheus-stack-kube-state-metrics-6c5d86887c-bzlfm   1/1     Running   0          30h
monitoring    kube-prometheus-stack-operator-5bbb5f4f64-bpvkv             1/1     Running   0          30h
monitoring    kube-prometheus-stack-prometheus-node-exporter-xt966        1/1     Running   0          30h
monitoring    prometheus-kube-prometheus-stack-prometheus-0               3/3     Running   0          28h
monitoring    thanos-compactor-674c68cfcc-9cwsd                           1/1     Running   0          5m31s
monitoring    thanos-query-65ff7b4f98-j2mc4                               1/1     Running   0          5m31s
monitoring    thanos-query-frontend-59df69d5c-nbhgh                       1/1     Running   0          5m31s
monitoring    thanos-ruler-0                                              1/1     Running   0          5m31s
monitoring    thanos-storegateway-0                                       1/1     Running   0          5m31s