php laravel lumen ELK日志采集系统安装

物料准备

ELK下载地址：下载中心 - Elastic 中文社区

elasticsearch-7.4.0-linux-x86_64.tar.gz

filebeat-7.4.0-linux-x86_64.tar.gz

logstash-7.4.0.tar.gz

kibana-7.4.0-linux-x86_64.tar.gz

filebeat_conf.yml

logstash-sample.conf

redis5.0.tar.gz

start_logstash.sh [脚本在文末]

start_filebeat.sh [脚本在文末]

start_es1.sh [脚本在文末]

start_es2.sh [脚本在文末]

start_es3.sh [脚本在文末]

安装准备

mkdir -p /data/soft /data/logs /data/bank  /data/elk

上传上述物料到 /data/soft 目录

安装JDK

1. 解压

cd /data/soft

tar zxf jdk-8u65-linux-x64.tar.gz -C /usr/local

2．配置环境变量

echo "export JAVA_HOME=/usr/local/jdk1.8.0_65/" >> /etc/profile

echo "export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar" >> /etc/profile

echo "export PATH=$JAVA_HOME/bin:$PATH" >> /etc/profile

ln -sf /usr/local/jdk1.8.0_65/bin/java /usr/bin/java

source /etc/profile

3．验证：

java -version

安装elasticsearch

cd /data/soft

tar zxf elasticsearch-7.4.0-linux-x86_64.tar.gz  -C /data/elk/

在一台服务器安装elasticsearch 3个节点的方法

1. 首先复制一个节点

cd /data/elk/

cp  -a elasticsearch-7.4.0  elasticsearch-7.4.0-node-1

cp  -a elasticsearch-7.4.0  elasticsearch-7.4.0-node-2

cp  -a elasticsearch-7.4.0  elasticsearch-7.4.0-node-3

rm –rf elasticsearch-7.4.0

cp /data/soft/start_es1.sh /data/elk/ elasticsearch-7.4.0-node-1

cp /data/soft/start_es2.sh /data/elk/ elasticsearch-7.4.0-node-2

cp /data/soft/start_es3.sh /data/elk/ elasticsearch-7.4.0-node-3

1. 修改node-2和node-3的配置文件

cd /data/elk/elasticsearch-7.4.0-node-2/config

cd /data/elk/elasticsearch-7.4.0-node-3/config

将elasticsearch.yml 中 node.master 设置为false 。node.master: false

1. 启动es。

cd /data/elk/elasticsearch-7.4.0-node-1

sh start_es1.sh start

cd /data/elk/elasticsearch-7.4.0-node-2

sh start_es2.sh start

cd /data/elk/elasticsearch-7.4.0-node-3

sh start_es3.sh start

4.  查看es是否启动成功

         ps -ef|grep elasticsearch

5. 查看es状态

         curl -i -XGET 'http://localhost:9200/_cluster/health?pretty'

查看是否启动成功： ps-ef|grep elasticsearch

   返回如下 表示启动成功

安装logstash

1. 解压

cd /data/soft

tar zxf  logstash-7.4.0.tar.gz  -C /data/elk/

1. 拷贝配置

cp /data/soft/logstash-sample.conf   /data/elk/logstash-7.4.0/config/

cp /data/soft/start_logstash.sh   /data/elk/logstash-7.4.0/

1. 启动

cd /data/elk/logstash-7.4.0/

sh start_logstash.sh start

1. 查看Logstash是否启动成功

        ps -ef|grep logstash

安装filebeat

1. 解压

cd /data/soft

tar zxf  filebeat-7.4.0-linux-x86_64.tar.gz  -C /data/elk/

1. 拷贝配置

cp /data/soft/filebeat_conf.yml   /data/elk/filebeat-7.4.0-linux-x86_64/

cp /data/soft/sstart_filebeat.sh   /data/elk/filebeat-7.4.0-linux-x86_64/

1. 启动

cd /data/elk/filebeat-7.4.0-linux-x86_64/

sh start_filebeat start

1. 查看filebeat是否启动成功

ps –ef|grep filebeat

安装kibana

1. 解压

cd /data/soft

tar zxf  kibana-7.4.0-linux-x86_64.tar.gz  -C /data/elk/

1. 修改配置(将页面显示改为中文)

cd /data/elk/kibana-7.4.0-linux-x86_64/config

将kibana.yml 中 i18n.locale: "en"  改为  i18n.locale: "zh-CN"

1. 启动

cd /data/elk/ kibana-7.4.0-linux-x86_64/

./bin/kibana – Q

1. 访问

http://127.0.0.1:5601

1. 设置kibana 使用用户名和密码。

使用nginx 做反向代理，然后在nginx 上生成密码。

密码生成：nginx控制kibana登录，设置用户名密码_xbttttt的博客-CSDN博客

Nginx配置如下：

upstream kibana_server {

    server 127.0.0.1:5601;

    ip_hash;

}

server {

    listen      80;

    server_name kibana.xxoooo.com;

    access_log  /data/logs/nginx/kibana.xxoooo.com.access.log;

    error_log   /data/logs/nginx/kibana.xxoooo.com.error.log;

    location / {

        proxy_redirect    off;

        auth_basic "Please input password";

        auth_basic_user_file /usr/local/nginx/conf/htpasswd.users;

        proxy_pass        http://kibana_server;

        proxy_pass_header User-Agent;

        proxy_set_header  Host $host;

        proxy_set_header  X-Real-IP $http_x_forwarded_for;

        proxy_set_header  X-Forwarded-For $http_x_forwarded_for;

    }

}

1. 配置索引后查看数据结果如下：

 

配置设置

Filebeat配置

1. 采集日志传输到logstash(/data/elk/filebeat-7.4.0-linux-x86_64/filebeat_conf.yml)

1. 输入配置

1. 输出配置

1. 采集日志传输到redis(/data/elk/filebeat-7.4.0-linux-x86_64/filebeat_conf.yml)

1. 输入配置

        

#平台用户中心日志
- type: log
  enabled: true
  paths:
    - /data/bank/xxxx/storage/logs/log*.log
  json:
    - keys_under_root: true
    - add_error_key: true
    - overwrite_keys: true
  tags: ["xxxx"]
  document_type: xxxx

1. 输出配置

output.redis:
   hosts: ["172.11.24.11:6379"]    #输出到redis的机器
   db: 2    #redis数据库的一个整数索引标识，redis不同于mysql有一个库的名字。redis总共0-15默认16个库。
   password: "ssfdsfdsfsss111"
   timeout: 5    #连接超时时间
   key: "default_list"    #以default_list的keys传输到redis

Logstash配置

1. 接收filebeat的日志(/data/elk/logstash-7.4.0/config/logstash-sample.conf)

1. Input输入配置

1. Output输出配置

output {
  #stdout { codec => rubydebug }
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "xxxooo.com-%{+YYYY.MM.dd}"
    #user => "elastic"
    #password => "changeme"
  }
}

1. Filter 配置

1. 接收redis的日志(/data/elk/logstash-7.4.0/config/logstash-sample.conf)

1. Input 输入配置

        

input {
  redis {
         data_type => "list"
         key => "default_list"
         host => "172.11.24.11"
         port => 6379
         db => 2
         threads => 1
         codec => json
         password => "13123111"
     }
}

1. Output 输出配置

        

if "xxxx" in [tags] {
          elasticsearch {
                hosts => ["http://localhost:9200"]
                index => "xxxx_log-%{+YYYY.MM.dd}"
                #user => "elastic"
                #password => "changeme"
          }
  }

1. Filter 配置

Elasticsearch 查看

1. 查看es中已有索引

curl localhost:9200/_cat/indices?v

 start_es1.sh 内容如下， start_es2.sh 、start_es3.sh  稍作修改即可。

#!/bin/bash -ile
start() {
  PID=`ps -ef|grep elasticsearch-7.4.0-node-1|grep -v grep|grep -v monitor|grep -v controller|awk '{print $2}'`
  if [ -z "$PID" ];then
    echo "start elasticsearch-7.4.0-node-1 process....."
    cd /data/elk/elasticsearch-7.4.0-node-1/
	nohup ./bin/elasticsearch > /data/logs/eslog-node-1.log 2>&1 &
	tail -f /data/logs/eslog-node-1.log
  else
    echo " elasticsearch-7.4.0-node-1 already runing.....PID=$PID"
  fi
}

stop() {
 PID=`ps -ef|grep elasticsearch-7.4.0-node-1|grep -v grep|grep -v monitor|grep -v controller|awk '{print $2}'`
 if [ x"$PID" != x ];then
   echo "kill elasticsearch-7.4.0-node-1 PID.....$PID"
   kill -9 $PID;
 fi
}


case $1 in
        start)
          start
        ;;
        stop)
          stop
        ;;
        restart)
          stop
          start
        ;;
        *)
                echo -e "no parameter"
        ;;
esac
exit 0

start_logstash.sh 命令如下：

#!/bin/bash -ile
start() {
  PID=`ps -ef|grep logstash-sample.conf|grep -v grep|grep -v monitor|awk '{print $2}'`
  if [ -z "$PID" ];then
    echo "start logstash process....."
    cd /data/elk/logstash-7.4.0/
    nohup ./bin/logstash -f config/logstash-sample.conf > /data/logs/logstash.log 2>&1 &
	tail -f /data/logs/logstash.log
  else
    echo " logstash already runing.....PID=$PID"
  fi
}

stop() {
 PID=`ps -ef|grep logstash-sample.conf|grep -v grep|grep -v monitor|awk '{print $2}'`
 if [ x"$PID" != x ];then
   echo "kill logstash PID.....$PID"
   kill -9 $PID;
 fi
}


case $1 in
        start)
          start
        ;;
        stop)
          stop
        ;;
        restart)
          stop
          start
        ;;
        *)
                echo -e "no parameter"
        ;;
esac
exit 0

start_filebeat.sh 命令如下：

#!/bin/bash -ile
start() {
  PID=`ps -ef|grep "filebeat_conf.yml" | grep -v grep|grep -v monitor | awk '{print $2}'`
  if [ -z "$PID" ];then
    echo "start process....."
    cd /data/elk/filebeat-7.4.0-linux-x86_64
    nohup ./filebeat -e -c filebeat_conf.yml > /data/logs/filebeat.log 2>&1 &
	tail -f /data/logs/filebeat.log
  else
    echo " already runing.....PID=$PID"
  fi
}

stop() {
 PID=`ps -ef|grep "filebeat_conf.yml" | grep -v grep|grep -v monitor | awk '{print $2}'`
 if [ x"$PID" != x ];then
   echo "kill PID.....$PID"
   kill -9 $PID;
 fi
}


case $1 in
        start)
          start
        ;;
        stop)
          stop
        ;;
        restart)
          stop
          start
        ;;
        *)
                echo -e "no parameter"
        ;;
esac
exit 0