demo-Prometheus+PushGateway+Grafana监控Flink on YARN作业

简述

flink metric用来对外暴露系统内部的一些运行指标，比如flink框架运行时的JVM相关配置，或者基于flink开发的项目。flink提供了Counter, Gauge, Histogram and Meter四种类型的指标。我们通过继承RichFunction拿到MetricGroup，并向其中填充指标。今天主要讲的是使用Prometheus+PushGateway+Grafana监控Flink on YARN作业

• 1.将 flink/opt/目录下的 flink-metrics-prometheus-1.9.1.jar 复制到lib下
  flink PrometheusPushGateway配置详情

[hadoop@hadoop001 lib]$ pwd
/home/hadoop/app/flink/lib
[hadoop@hadoop001 lib]$ ll
-rw-r--r-- 1 hadoop hadoop    103761 Apr 28 15:38 flink-metrics-prometheus-1.9.1.jar

# 配置flink-conf.yaml,进入/home/hadoop/app/flink/conf目录
[hadoop@hadoop001 conf]$ pwd
/home/hadoop/app/flink/conf

metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
# 这里写PushGateway的主机名与端口号
metrics.reporter.promgateway.host: hadoop001
metrics.reporter.promgateway.port: 9091
# Flink metric在前端展示的标签（前缀）与随机后缀
metrics.reporter.promgateway.jobName: flink-metrics
metrics.reporter.promgateway.randomJobNameSuffix: true
metrics.reporter.promgateway.deleteOnShutdown: false

• 2.下载 Prometheus PushGateway ,解压即可使用
  网址: https://prometheus.io/

wget https://github.com/prometheus/prometheus/releases/download/v2.18.0-rc.0/prometheus-2.18.0-rc.0.linux-amd64.tar.gz
# 注意:pushgateway 版本有可能与flink不兼容 ,请自行选择合适的版本
wget https://github.com/prometheus/pushgateway/releases/download/v1.2.0/pushgateway-1.2.0.linux-amd64.tar.gz

• 3.配置 prometheus 的prometheus.yml文件 scrape_configs

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
    - targets: ['hadoop001:9090']
    #注意格式 空格
  - job_name: 'pushgateway'
    static_configs:
      - targets: ['hadoop001:9091']
        labels:
          instance: 'pushgateway'

• 4.启动 页面查看是否启动成功 ,监控

nohup ./pushgateway --web.listen-address :9091 > /var/log/pushgateway.log 2>&1 &

nohup ./prometheus --config.file=prometheus.yml > /var/log/prometheus.log 2>&1 &
# 端口号的查看
	netstat -apn | grep -E '9091|3000|9090'

• 5.Grafana的安装,展示监控数据
  网址: https://grafana.com/

wget https://dl.grafana.com/oss/release/grafana-6.7.3.linux-amd64.tar.gz
tar -zxvf grafana-6.7.3.linux-amd64.tar.gz

添加数据源 ,指定要监控的指标 以及添加相关报警规则等

• 6.运行flink example ,查看是否监控成功

nc -l 19999
./bin/flink run -m yarn-cluster -p 4 -yjm 1024m -ytm 1024m --yarnname "window-wordcount" ./examples/streaming/SocketWindowWordCount.jar --hostname hadoop001 --port 19999