ELK实战--分析MySQL慢查询日志并生成视图

一、背景

1.MySQL慢查询日志格式：

# Time: 181109 15:04:08
# User@Host: t***[t***] @  [172.16.14.51]  Id: 8960747
# Query_time: 35.918265  Lock_time: 0.000141 Rows_sent: 1  Rows_examined: 11699162
SET timestamp=1541747048;
select count(*) from trade_risk_control_record

2.MySQL慢查询日志已通过rsyslog实时传输到logstash作为Indexer的节点。

二、logstash配置文件

input部分

input {
  file {
    type => "logstash-rc-mysql-slow"
        path => "/opt/data/logs/localhost-172.16.14.35/db1-slow.log"
        codec => multiline {
          pattern => "^# Time:"
          negate => true
          what => "previous"
        }
        stat_interval => 1
                discover_interval => 1
                start_position=>"end"
                sincedb_path => "/dev/null"
  }
}

filter部分

if [type] == "logstash-rc-mysql-slow" {
  grok {
    patterns_dir => ["/usr/local/logstash/etc/conf.d/patterns/mysql"]
    match => { "message" => "%{LONGQUERYLOG}" }
  }
  date {
    match => ["timestamp","UNIX"]
  }
  mutate {
    convert => [ "query_time", "float" ]
    convert => [ " lock_time", "float" ]
    remove_field => "message"
    remove_field => "timestamp"
       }
  } 

output部分

if [type] == "logstash-rc-mysql-slow" {
             elasticsearch {
                            hosts => ["172.16.1.25","172.16.1.26","172.16.1.27"]
                            index => 'logstash-mysql_slow_log-%{+YYYY-MM-dd}'
                            codec=>plain{charset=>"UTF-8"}
                          }
        }

patterns部分

LONGQUERYLOG ^#\s+Time:.*\n#\s+User@Host:\s+%{USER:user}\[[^\]]+\]\s+@\s+(?:(?\S*) )?\[(?:%{IP:clientip})?\]\s+Id:\s+%{NUMBER:id}\n# Query_time: %{NUMBER:query_time}\s+Lock_time: %{NUMBER:lock_time}\s+Rows_sent: %{NUMBER:rows_sent}\s+Rows_examined: %{NUMBER:rows_examined}\nSET\s+timestamp=%{NUMBER:timestamp};\n(?[\s\S]*)

三、kibana展示

1.创建索引

2.发现数据

包括字段：

3.绘制visualize
例1：统计数量排名前10的sql语句及对应的查询时间