Fluentd+ElasticSearch+Kibana 搭建日誌類文件分析器 安裝篇

一 查看當前機器信息

1. 操作系統

[coolbeans@coolbeans-fluentd ~]$ cd /proc
[coolbeans@coolbeans-fluentd proc]$ lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID:    CentOS
Description:    CentOS release 6.4 (Final)
Release:    6.4
Codename:    Final

2. hostname

[coolbeans@coolbeans-fluentd ~]$ cat /proc/sys/kernel/hostname
coolbeans-fluentd
[coolbeans@coolbeans-fluentd ~]$ hostname
coolbeans-fluentd

3. 内存

[coolbeans@coolbeans-fluentd ~]$ free
             total       used       free     shared    buffers     cached
Mem:       1922464    1067420     855044          0     256628     230204
-/+ buffers/cache:     580588    1341876
Swap:      4128760          0    4128760
[coolbeans@coolbeans-fluentd ~]$ cat /proc/meminfo
MemTotal:        1922464 kB
MemFree:          854912 kB
Buffers:          256628 kB
Cached:           230204 kB
SwapCached:            0 kB
Active:           690624 kB
Inactive:         193840 kB
Active(anon):     397784 kB
Inactive(anon):        4 kB
Active(file):     292840 kB
Inactive(file):   193836 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       4128760 kB
SwapFree:        4128760 kB
Dirty:                16 kB
Writeback:             0 kB
AnonPages:        397540 kB
Mapped:            26532 kB
Shmem:               164 kB
Slab:             150432 kB
SReclaimable:      91164 kB
SUnreclaim:        59268 kB
KernelStack:        1528 kB
PageTables:         4836 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     5089992 kB
Committed_AS:     645828 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      274520 kB
VmallocChunk:   34359447820 kB
HardwareCorrupted:     0 kB
AnonHugePages:    321536 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        8192 kB
DirectMap2M:     2088960 kB

4. CPU

[coolbeans@coolbeans-fluentd ~]$ cat /proc/cpuinfo | grep 'model name' |uniq
model name    : Intel(R) Xeon(R) CPU           X5660  @ 2.80GHz

5. Disk

[coolbeans@coolbeans-fluentd /]$ df -h
文件系统          容量  已用  可用 已用%% 挂载点
/dev/mapper/vg_coolbeansfluentd-lv_root
                       50G  3.1G   44G   7% /
tmpfs                 939M     0  939M   0% /dev/shm
/dev/sda1             485M   64M  397M  14% /boot
/dev/mapper/vg_coolbeansfluentd-lv_home
                      439G  1.6G  415G   1% /home

二 查看Max # of File Descriptors

    ulimit -n

三 安裝git

     安裝其所依賴之包：

     yum install zlib-devel
yum install openssl-devel
yum install perl
yum install cpio
yum install expat-devel
yum install gettext-devel 

yum install autoconf

//安裝GIT

wget http://www.codemonkey.org.uk/projects/git-snapshots/git/git-latest.tar.gz
tar xzvf git-latest.tar.gz
cd git-{date}
autoconf
./configure --with-curl=/usr/local
make 

make install

四 安裝RUBY
下載ruby-2.0.0-p247.tar.gz
tar -zxvf ruby-2.0.0-p247.tar.gz
cd ruby-2.0.0-p247
./configure  -prefix=/opt/modules/ruby
make
make install
添加環境變量：export PATH=/opt/modules/ruby/bin:$PATH

測試RUBY GEM:
[coolbeans@coolbeans-fluentd ~]$ gem -v
2.0.3
備註：
在安裝RUBY前請確保OPEN-SSL已安裝
五 安裝elasticsearch
下載：elasticsearch-0.90.0.RC2.tar.gz
tar -zxvf elasticsearch-0.90.0.RC2.tar.gz
cd elasticsearch-0.90.0.RC2
啟動： ./bin/elasticsearch -f

六 安裝bundle
[coolbeans@coolbeans-fluentd bin]$ sudo ./gem install bundle
[sudo] password for coolbeans:
Fetching: bundler-1.3.5.gem (100%)
Successfully installed bundler-1.3.5
Fetching: bundle-0.0.1.gem (100%)
Successfully installed bundle-0.0.1
Parsing documentation for bundler-1.3.5
Installing ri documentation for bundler-1.3.5
Parsing documentation for bundle-0.0.1
Installing ri documentation for bundle-0.0.1
2 gems installed

七 安裝Kibana
[coolbeans@coolbeans-fluentd ~]$ cd Kibana-0.2.0
[coolbeans@coolbeans-fluentd Kibana-0.2.0]$ bundle install --path vendor/bundle
Fetching gem metadata from http://rubygems.org/.........
Fetching gem metadata from http://rubygems.org/..
Resolving dependencies...
Installing rake (10.1.0)
Installing daemons (1.1.9)
Installing diff-lcs (1.1.3)
Installing eventmachine (1.0.0)
Installing fastercsv (1.5.5)
Installing json (1.7.5)
Installing rack (1.4.1)
Installing rack-protection (1.2.0)
Installing tilt (1.3.3)
Installing sinatra (1.3.3)
Installing thin (1.5.0)
Installing tzinfo (0.3.35)
Using kibana (0.0.1) from source at .
Installing rspec-core (2.11.1)
Installing rspec-expectations (2.11.3)
Installing rspec-mocks (2.11.3)
Installing rspec (2.11.0)
Using bundler (1.3.5)
Your bundle is complete!
It was installed into ./vendor/bundle

啟動Kibana:

bundle exec ruby kibana.rb

默認新動態下，只可在本地訪問：

修改 KibanaConfig.rb  16行. 將 127.0.0.1 修改為 0.0.0.0 以綁定所有IP

# The adress ip Kibana should listen on
KibanaHost = '127.0.0.1'

八 安裝Fluented RPM:
執行該腳本即可
http://toolbelt.treasure-data.com/sh/install-redhat.sh

九 安裝ElasticSearch plugin for Fluentd: fluent-plugin-elasticsearch

/usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch

[coolbeans@coolbeans-fluentd bin]$ sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-elasticsearch
[sudo] password for coolbeans:
Fetching: fluent-plugin-elasticsearch-0.1.2.gem (100%)
Successfully installed fluent-plugin-elasticsearch-0.1.2
1 gem installed
Installing ri documentation for fluent-plugin-elasticsearch-0.1.2...
Installing RDoc documentation for fluent-plugin-elasticsearch-0.1.2...

修改 /etc/td-agent/td-agent.conf為：

  type syslog
  port 42185
  tag syslog



  type forward



  type elasticsearch
  logstash_format true
  flush_interval 10s # for testing

啟動td-agent

sudo /etc/init.d/td-agent start

十 fluent-plugin-notifier的安裝
     學前最新版本為：0.2.1
sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-notifier -v 0.2.1

十一  fluent-plugin-mail的安裝
sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-mail

十二 fluent-plugin-tail-ex安裝
在in_tail plugin 的基礎上擴展，其主要新增功能為支持目錄文件收集
安裝命令：sudo /usr/lib64/fluent/ruby/bin/fluent-gem install fluent-plugin-tail-ex

十二  rsyslogd
系統已自帶