k8s日志预研—收集Kubernetes event（容器日志）

以下内容根据简书链接，并结合自己的日志收集系统整理而成

（一）使用收集event方案

开源项目eventrouter，地址为：https://github.com/heptiolabs/eventrouter

（二）收集流程

大概流程为：
1、启动eventrouter容器,挂载/data/log/eventrouter目录
2、启动filebeat容器,挂载/data/log/eventrouter目录
3、filebeat收集/data/log/eventrouter目录下的日志
4、filebeat数据发送到logstash进行处理
5、logstash处理后的数据存储到Elasticsearch
6、kibana可视化平台进行索引展示

（三）测试集群

IP	角色
10.0.3.239	master
10.0.3.247	node
10.0.3.248	node
10.0.3.104	logstash、es、kibana

k8s笔记8-部署filebeat+ELK日志方案，参考该博客部署服务器10.0.3.104上的ELK环境。

小福利：

由于elasticsearch-6.3.2.rpm、kibana-6.3.2-x86_64.rpm外网下载很慢，分享2个地址，大家可以在迅雷里面下载，节约时间：
kibana地址：https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-x86_64.rpm
elasticsearch地址：https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.rpm
logstash地址：各个版本的链接查看地址：https://www.elastic.co/downloads/past-releases?page=6#logstash
下载地址：https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.rpm
安装RPM的命令：

rpm -ivh *.rpm --force --nodeps

（四）修改过的eventrouter配置文件，cat filebeat-eventrouter.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: eventrouter 
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: eventrouter 
rules:
- apiGroups: [""]
  resources: ["events"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: eventrouter 
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: eventrouter
subjects:
- kind: ServiceAccount
  name: eventrouter
  namespace: kube-system
---
apiVersion: v1
data:
  config.json: |- 
    {
      "sink": "glog"
    }
kind: ConfigMap
metadata:
  name: eventrouter-cm
  namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |-
    filebeat.prospectors:
    - input_type: log
      paths:
        - "/data/log/eventrouter/*"
      fields:
        app: www
        type: eventrouter-log
      fields_under_root: true
      multiline:
        pattern: '^\['
        negate: true
        match: after
    output.logstash:
      hosts: ["10.0.3.104:5044"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: eventrouter
  namespace: kube-system
  labels:
    app: eventrouter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: eventrouter
  template:
    metadata:
      labels:
        app: eventrouter
        tier: control-plane-addons
    spec:
      containers:
        - name: kube-eventrouter
          image: baiyongjie/eventrouter:v0.2
          imagePullPolicy: IfNotPresent
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "/eventrouter -v 3 -log_dir /data/log/eventrouter"
          volumeMounts:
          - name: eventrouter-cm
            mountPath: /etc/eventrouter
          - name: log-path
            mountPath: /data/log/eventrouter
        - name: filebeat
          image: test.com/project/filebeat:6.3.2
          command:
            - "/bin/sh"
          args:
            - "-c"
            - "filebeat -c /etc/filebeat/filebeat.yml"
          volumeMounts:
          - name: filebeat-config
            mountPath: /etc/filebeat/
          - name: log-path
            mountPath: /data/log/eventrouter
      serviceAccount: eventrouter
      volumes:
        - name: eventrouter-cm
          configMap:
            name: eventrouter-cm
        - name: filebeat-config
          configMap:
            name: filebeat-config
        - name: log-path
          emptyDir: {}

修改/etc/logstash/conf.d/logstash-to-es.conf文件，添加logstash的内容：

        else if [type] == "eventrouter-log" {
           elasticsearch {
              hosts => ["http://127.0.0.1:9200"]
              index => "eventrouter-log-%{+YYYY.MM.dd}"
           }
        }

（五）在kibana的页面展示添加index Patterns，展示

在服务器中查看即时的event的语句为

kubectl -n kube-system describe pod grafana-0 | grep -A 10 Events:

当集群中有pod的contrainer重启或者出错时，会在该界面看到对应时间戳的日志展示，方便与获取容器日志与应用日志之间的匹配关系。

后续需要加的：
将日志内容进行持久化存储