Logstash用法实例
首先提供一些参考的网站：
https://www.gitbook.com/book/chenryn/kibana-guide-cn
http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html
https://www.gitbook.com/book/chenryn/kibana-guide-cn
http://www.learnes.net/index.html
下面由简单到复杂，介绍一些Logstash的配置方法
安装完Logstash后，默认安装在/opt/logstash中，在bin目录中有一些命令可以使用。
最简单的配置语法：
input {
stdin {}
syslog {}
}
Logstash的Helloworld：
[root@BLELOIST003 logstash]# bin/logstash -e 'input{stdin{}}output{stdout{}}'
!!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
Settings: Default pipeline workers: 8
Logstash startup completed
hello world
2016-03-17T03:30:02.147Z BLELOIST003.lenovo.com hello world
[root@BLELOIST003 logstash]#
[root@BLELOIST003 logstash]# bin/logstash -e 'input{stdin{}}output{stdout{codec=>rubydebug}}'
!!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
Settings: Default pipeline workers: 8
Logstash startup completed
hello world
{
"message" => "hello world",
"@version" => "1",
"@timestamp" => "2016-03-17T03:31:30.308Z",
"host" => "BLELOIST003.lenovo.com"
}
最基本的配置组成是input output，可选的是filter。input中的stdin和output中的stdout代表了命令行窗口的输入输出。
监控apache日志，这里没有做过滤。
[root@BLELOIST003 logstash]# bin/logstash -e '
> input {
> file {
> type => "apache-access"
> path => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log*"
> start_position => end
> sincedb_path => "/dev/null"
> }
> }
> filter{
>
> }
> output {
> stdout{ codec=>rubydebug }
> }'
!!! Please upgrade your java version, the current version '1.7.0_45-mockbuild_2013_10_22_03_37-b00' may cause problems. We recommend a minimum version of 1.7.0_51
Settings: Default pipeline workers: 8
Logstash startup completed
{
"message" => "10.100.90.241 - - [17/Mar/2016:11:35:00 +0800] \"GET / HTTP/1.1\" 304 - 411",
"@version" => "1",
"@timestamp" => "2016-03-17T03:35:01.162Z",
"path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
"host" => "BLELOIST003.lenovo.com",
"type" => "apache-access"
}
{
"message" => "10.99.80.8 - dummy002 [17/Mar/2016:11:35:01 +0800] \"POST /LOIS/servlet/SEOTranslationService HTTP/1.0\" 200 90360 503291",
"@version" => "1",
"@timestamp" => "2016-03-17T03:35:03.178Z",
"path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
"host" => "BLELOIST003.lenovo.com",
"type" => "apache-access"
}
start_position => end 代表了只监控新增加的日志内容。
加上正则表达式的过滤：
[root@BLELOIST003 logstash]# bin/logstash -e '
> input {
> file {
> type => "apache-access"
> path => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log*"
> start_position => end
> sincedb_path => "/dev/null"
> }
> }
> filter{
> if [path] =~ "access" {
> mutate { replace => { "type" => "ApacheLogs" } }
> grok {
> match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:userName} \[%{HTTPDATE:logTime}\] \"%{WORD:verb} %{NOTSPACE:request} (?:%{NOTSPACE:httpversion}|)\" (?:%{NUMBER:state}|-) (?:%{NUMBER:bytes}|-) %{NUMBER:duration}"}
> }
> }
> date {
> match => [ "logTime" , "dd/MMM/yyyy:HH:mm:ss Z" ]
> }
> }
> output {
> stdout{ codec=>rubydebug }
> }'
Settings: Default pipeline workers: 8
Logstash startup completed
{
"message" => "10.100.90.241 - - [17/Mar/2016:12:05:27 +0800] \"GET /Windchill/ HTTP/1.1\" 404 208 424",
"@version" => "1",
"@timestamp" => "2016-03-17T04:05:27.000Z",
"path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
"host" => "BLELOIST003.lenovo.com",
"type" => "ApacheLogs",
"clientip" => "10.100.90.241",
"ident" => "-",
"userName" => "-",
"logTime" => "17/Mar/2016:12:05:27 +0800",
"verb" => "GET",
"request" => "/Windchill/",
"httpversion" => "HTTP/1.1",
"state" => "404",
"bytes" => "208",
"duration" => "424"
}
{
"message" => "10.100.90.241 - - [17/Mar/2016:12:05:27 +0800] \"GET /favicon.ico HTTP/1.1\" 200 207 476",
"@version" => "1",
"@timestamp" => "2016-03-17T04:05:27.000Z",
"path" => "/apphome/ptc/Windchill_10.2/HTTPServer/logs/access.log_2016-03-17_00_00_00",
"host" => "BLELOIST003.lenovo.com",
"type" => "ApacheLogs",
"clientip" => "10.100.90.241",
"ident" => "-",
"userName" => "-",
"logTime" => "17/Mar/2016:12:05:27 +0800",
"verb" => "GET",
"request" => "/favicon.ico",
"httpversion" => "HTTP/1.1",
"state" => "200",
"bytes" => "207",
"duration" => "476"
}
这里的if [path] =~ "access"代表了判断语句，可以参考这里：
http://kibana.logstash.es/content/logstash/get_start/full_config.html
这里的过滤中grok中匹配：
match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:userName} \[%{HTTPDATE:logTime}\] \"%{WORD:verb} %{NOTSPACE:request} (?:%{NOTSPACE:httpversion}|)\" (?:%{NUMBER:state}|-) (?:%{NUMBER:bytes}|-) %{NUMBER:duration}"}
其中%{IPORHOST:clientip}表示具体的正则匹配，IPORHOST代表了logstash的pattern，具体可以参考如下链接，而clientip则代表了在logstash中信息传递的变量。
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns