Filebeat7.17收集SpringBoot应用k8s容器日志
版本
SpringBoot 2.6.6
Filebeat7.17.6
Kafka_2.13-2.8.2
ELK7.17.6
应用场景
springboot应用通过Deployment方式部署到k8s集群中，收集springboot应用日志。
关键实现
在filebeat配置文件中，采用filebeat.autodiscover方式，采集容器日志；通过template+condition方式，对k8s namespace进行过滤，只采集特定的namespace；根据namespace将采集的日志发送到kafka的不同topic；通过processors:script对采集的kubernetes信息进行聚合；通过drop_fields删除无关字段。
在deployment.yaml中，通过k8s Downward API获取Pod name和namespace注入到环境变量；通过subPathExpr配置不同应用的日志目录。
SpringBoot日志配置
https://docs.spring.io/spring-boot/docs/current/reference/html/howto.html#howto.logging
根据配置方式的不同，springboot应用日志可以输出到控制台或日志文件。
1、只输出到控制台
不做任何配置，默认情况下日志只输出到控制台。
2、只输出到文件
src/main/resources/logback-spring.xml：
<configuration>
<include resource="org/springframework/boot/logging/logback/defaults.xml" />
<property name="LOG_FILE" value="${LOG_FILE:-${LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/}spring.log}"/>
<include resource="org/springframework/boot/logging/logback/file-appender.xml" />
<root level="INFO">
<appender-ref ref="FILE" />
root>
configuration>
3、同时输出到控制台和文件
默认情况下日志会输出到控制台，输出到文件可以在application.properties文件中配置日志级别和日志文件名称
logging.file.name=/var/appdeploy/appdeploy.log
更复杂的配置需要在xml日志配置文件中完成。
日志收集架构
数据流向
控制台日志输出：Pod -> /var/log/containers/.log -> Filebeat -> Kafka -> Logstash -> Elasticsearch -> Kibana
文件日志输出：Pod -> /logs//*.log -> Filebeat -> Kafka -> Logstash -> Elasticsearch -> Kibana
文件日志输出，在创建Deployment的时候，通过hostpath写到主机的/logs/{应用名称}/目录下。
部署方式
采用Daemonset方式部署轻量级日志收集程序Filebeat，对节点的/var/logs/containers/目录及 /logs/*/目录进行日志采集。
ELK采用elasticsearch-7.17.6-linux-x86_64.tar.gz等tar包方式安装，详见《ELK7安装》。
Kafka采用集群方式安装，详见《kafka2.8.2集群安装》。
Filebeat配置
部署2套filebeat，将容器日志采集和文件日志采集的filebeat分开部署，因为测试时发现在一个filebeat配置文件中同时配置autodiscover和filebeat.inputs(type: filestream)，filebeat.inputs并没有被采集。(也可能是配置有误，期待高手指导)
- 采用filebeat.autodiscover方式，采集容器日志。
- 通过template+condition方式，对k8s namespace进行过滤，只采集特定的namespace。
- 根据namespace将采集的日志发送到kafka的不同topic。
- 通过processors:script对采集的kubernetes信息进行聚合。
- 通过drop_fields删除无关字段。
filebeat部署文件如下所示：
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
spec:
selector:
matchLabels:
k8s-app: filebeat
template:
metadata:
labels:
k8s-app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.17.6
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0640
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
# When filebeat runs as non-root user, this directory needs to be writable by group (g+w).
path: /var/lib/filebeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: filebeat
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: filebeat
namespace: kube-system
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: Role
name: filebeat
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: filebeat-kubeadm-config
namespace: kube-system
subjects:
- kind: ServiceAccount
name: filebeat
namespace: kube-system
roleRef:
kind: Role
name: filebeat-kubeadm-config
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: filebeat
labels:
k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
verbs:
- get
- watch
- list
- apiGroups: ["apps"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat
# should be the namespace where filebeat is running
namespace: kube-system
labels:
k8s-app: filebeat
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs: ["get", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: filebeat-kubeadm-config
namespace: kube-system
labels:
k8s-app: filebeat
rules:
- apiGroups: [""]
resources:
- configmaps
resourceNames:
- kubeadm-config
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: filebeat
namespace: kube-system
labels:
k8s-app: filebeat
---
filebeat容器采集配置文件：
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
labels:
k8s-app: filebeat
data:
filebeat.yml: |-
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
hints.default_config.enabled: false
templates:
- condition:
and:
- not:
equals:
kubernetes.namespace: kube-system
- not:
equals:
kubernetes.namespace: ceph-csi
- not:
equals:
kubernetes.namespace: tekton-pipelines
config:
- type: container
paths:
- /var/log/containers/*-${data.kubernetes.container.id}.log
processors:
- script:
lang: javascript
source: >
function process(event){
var k8s = event.Get("kubernetes");
var k8sInfo = {
namespace: k8s.namespace,
podName: k8s.pod.name,
hostName: k8s.node.hostname,
containerName: k8s.container.name,
labels: k8s.labels,
k8sName: "cicd"
}
event.Put("k8s", k8sInfo);
}
- drop_fields:
fields: ["host", "ecs", "log", "agent", "input", "tags", "container", "stream", "orchestrator", "kubernetes"]
ignore_missing: true
output.kafka:
hosts: ["27.196.38.200:9092", "27.196.38.201:9092", "27.196.38.202:9092"]
topic: '%{[k8s.k8sName]}-%{[k8s.namespace]]}'
partition.round_robin:
reachable_only: false
required_acks: 1
keep-alive: 120
max_message_bytes: 1000000
filebeat文件采集配置文件：
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config-app
namespace: kube-system
labels:
k8s-app: filebeat-app
data:
filebeat.yml: |-
filebeat.inputs:
- type: filestream
id: log-input
paths:
- "/logs/*/*.log"
parsers:
- multiline:
type: pattern
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after
processors:
- script:
lang: javascript
source: >
function process(event){
var path = event.Get("log.file.path");
if(path !== null){
var pathList = path.split("/");
if(pathList.length > 2){
var subDir = pathList[2];
var nsPod = subDir.split("_");
var k8sInfo = {
namespace: nsPod[0],
podName: nsPod[1],
containerName: nsPod[2],
k8sName: "cicd"
};
event.Put("k8s", k8sInfo);
}
}
}
- drop_fields:
fields: ["host", "ecs", "agent", "input"]
ignore_missing: true
output.kafka:
hosts: ["27.196.38.200:9092", "27.196.38.201:9092", "27.196.38.202:9092"]
topic: '%{[k8s.k8sName]}-%{[k8s.namespace]}'
partition.round_robin:
reachable_only: false
required_acks: 1
keep-alive: 120
max_message_bytes: 1000000
Logstash配置
input {
kafka {
bootstrap_servers => "27.196.38.200:9092,27.196.38.201:9092,27.196.38.202:9092"
codec => json {
charset => "UTF-8"
}
topics_pattern => "cicd-.*"
decorate_events => "basic"
auto_offset_reset => "latest"
group_id => "cicd_log"
client_id => "logstash"
consumer_threads => 3
}
}
output {
elasticsearch {
hosts => ["http://27.196.38.200:9200","http://27.196.38.201:9200","http://27.196.38.202:9200"]
index => "%{[k8s][k8sName]}-%{[k8s][namespace]}-%{+YYYY.MM.dd}"
}
}
在kibana中查看日志内容：
Deployment配置
通过k8s Downward API获取Pod name和namespace注入到环境变量中。
通过subPathExpr配置不同应用的日志目录。
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
k8s.deploy.cn/appId: "1"
k8s.deploy.cn/name: appdeloy
name: appdeloy
namespace: default
spec:
replicas: 1
selector:
matchLabels:
k8s.deploy.cn/appId: "1"
k8s.deploy.cn/name: appdeloy
template:
metadata:
labels:
k8s.deploy.cn/appId: "1"
k8s.deploy.cn/name: appdeloy
spec:
containers:
- env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: appdeploy:v1
imagePullPolicy: IfNotPresent
name: appdeloy
volumeMounts:
- mountPath: /var/appdeploy/
name: app-log
subPathExpr: $(NAMESPACE)_$(POD_NAME)_appdeloy
volumes:
- hostPath:
path: /logs
type: "DirectoryOrCreate"
name: app-log