promtail + loki + grafana实现日志服务
介绍
promtail 部署在日志被收集端
loki 日志服务 ip: 192.168.0.39
grafana 日志展示查看
docker部署loki
目录：
/data/loki_dir/data 数据目录（索引啥的）
/data/loki_dir/config 配置文件目录
权限：
chmod 777 /data/loki_dir/data配置文件
# cat config/loki-local-config.yaml
auth_enabled: false
server:
http_listen_port: 3100
grpc_listen_port: 9096
ingester:
wal:
enabled: true
dir: /data/loki/wal
lifecycler:
address: 127.0.0.1
ring:
kvstore:
store: inmemory
replication_factor: 1
final_sleep: 0s
chunk_idle_period: 1h # Any chunk not receiving new logs in this time will be flushed
max_chunk_age: 1h # All chunks will be flushed when they hit this age, default is 1h
chunk_target_size: 1048576 # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
chunk_retain_period: 30s # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
max_transfer_retries: 0 # Chunk transfers disabled
schema_config:
configs:
- from: 2022-06-06
store: boltdb-shipper
object_store: filesystem
schema: v11
index:
prefix: index_
period: 24h
storage_config:
boltdb_shipper:
active_index_directory: /data/loki/boltdb-shipper-active
cache_location: /data/loki/boltdb-shipper-cache
cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
shared_store: filesystem
filesystem:
directory: /data/loki/chunks
compactor:
working_directory: /data/loki/boltdb-shipper-compactor
shared_store: filesystem
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
chunk_store_config:
max_look_back_period: 0s
table_manager:
retention_deletes_enabled: false
retention_period: 0s如果你配置了多个promtail终端，然后同时启动，日志量很大，可能会报错，好像是419，反正意思就是loki有限制，你发送的日志超过限制了。
解决方法：
上面的配置文件，增加最后两个限制
limits_config:
reject_old_samples: true
reject_old_samples_max_age: 168h
# 每个用户每秒的采样率限制
ingestion_rate_mb: 60
# 每个用户允许的采样突发大小
ingestion_burst_size_mb: 80docker-compose文件
# cat docker-compose.yml
version: '3'
services:
loki:
image: grafana/loki:2.2.0
restart: always
volumes:
- /data/loki_dir/data:/data/loki/
- /data/loki_dir/config/loki-local-config.yaml:/etc/loki/local-config.yaml
environment:
TZ: "Asia/Shanghai"
ports:
- "0.0.0.0:3100:3100"启动
docker-compose up -dpromtail部署
选择一个版本
https://github.com/grafana/loki/tags
下载 promtail-linux-adm64.zip， 找个你喜欢的地方解压
编写配置文件
# cat /data/promtail/promtail-config.yml
server:
http_listen_port: 9000
grpc_listen_port: 0
positions:
filename: /etc/promtail/positions.yaml # 游标记录上一次同步位置
clients:
- url: http://192.168.0.39:3100/loki/api/v1/push #将日志发送给loki服务
scrape_configs:
- job_name: free-study-pro2 #名称
static_configs:
- targets:
- localhost
labels: #标签，后期grafana通过这些找到对应的日志
job: free-study-pro2
app: free-study-pro2-VM_0_21_centos
__path__: /data/free_study_pro2/log.out # 收集的日志文件创建同步位置记录文件目录
mkdir /etc/promtail启动
nohup /data/promtail/promtail-linux-amd64 -config.file=/data/promtail/promtail-config.yml >/dev/null 2>&1 &第一次启动可以前台启动，看看有没有报错啥的
/data/promtail/promtail-linux-amd64 -config.file=/data/promtail/promtail-config.ymlgrafana配置
日志部分的图没截，怕看，柱状图中INFO日志是绿色、红色是ERROR日志
最后使用prometheus集成blackbox_exporter组件对promtail进行tcp端口监控告警，避免意外停止不知道
# TCP 端口监测
- job_name: "blackbox_telnet_port"
scrape_interval: 60s
metrics_path: /probe
params:
module: [tcp_connect]
static_configs:
- targets: [ '172.30.0.2:9000' ]
labels:
tcp_name: 'promtail'
- targets: [ '172.30.0.3:9000' ]
labels:
tcp_name: 'promtail'
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 192.168.0.39:9300prometheus集成blackbox_exporter组件文档参考：
prometheus监控传统环境监控（三）URL监控_我的喵叫初六的博客-CSDN博客