如何收集k8s pod的服务日志（rancher）

一、环境情况说明 

当前环境是k8s+rancher+filebeat+es+kibana 本文只讲解部署filebeat 收集容器日志的过程

、使用daemonset的方式部署filebeat，es+kibana+k8s已提前创建好的

二、部署安装

1、在已创建完成的k8s环境中，用kubectl先创建一个命名空间elk-log，新增一个filebeat目录,上传一下4个代码文件

filebeat.daemonset.yml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: elk-log
  name: filebeat
  labels:
    app: filebeat
spec:
  selector:
    matchLabels:
      app: filebeat
  template:
    metadata:
      labels:
        app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 300
      containers:
      - name: filebeat
        image: xx.xx.cn/duo-prod/filebeat:7.8.0 #更改为可用的filebeat镜像地址
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: filebeat-indice-lifecycle
          mountPath: /etc/indice-lifecycle.json
          readOnly: true
          subPath: indice-lifecycle.json
        - name: data
          mountPath: /usr/share/filebeat/data

 filebeat.indice-lifecycle.configmap.yml

---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: elk-log
  name: filebeat-indice-lifecycle
  labels:
    app: filebeat
data:
  indice-lifecycle.json: |-
    {
      "policy": {
        "phases": {
          "hot": {
            "actions": {
              "rollover": {
                "max_size": "5GB" ,
                "max_age": "1d"
              }
            }
          },
          "delete": {
            "min_age": "15d",
            "actions": {
              "delete": {}
            }
          }
        }
      }
    }
~
~

filebeat.permission.yml

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: elk-log
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    app: filebeat
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: elk-log
  name: filebeat
  labels:
    app: filebeat
~

filebeat.settings.configmap.yml

filebeat.inputs:
- type: container
  enabled: true
  paths:
  - /var/log/containers/*.log  #正则匹配你想收集的日志
  processors:
  - add_kubernetes_metadata:
      default_indexers.enabled: true
      in_cluster: true
      matchers:
      - logs_path:
          logs_path: "/var/log/containers/"
  - add_cloud_metadata:
output.elasticsearch:
  hosts: ["xx.xx.xx.xx:9200"] #es地址，集群用","隔开
  username: "elastic"
  password: "xxxxxxxx"
  indices:
  - index: "k8s-%{[kubernetes.namespace]}.%{[kubernetes.container.name]}-%{+YYYY.MM.dd}.log"

2、执行部署安装  

kubectl apply -f filebeat/filebeat.daemonset.yml
kubectl apply -f filebeat.indice-lifecycle.configmap.yml
kubectl apply -f filebeat.permission.yml
kubectl apply -f filebeat.settings.configmap.yml

完成后rancher查看pod以及configmap

3、kibana 查看索引是否已经获取到