安装X-Pack插件之后Logstash无法连接Elasticsearch

为了使用X-Pack强大的特性，我分别在Kibana, Logstash和Elasticsearch根目录中安装了X-Pack插件，安装完成重启各软件之后，满心欢喜。可是随后令人蛋痛的事情发生了，Logstash不能正常工作。
查看日志文件：/var/log/logstash/logstash-plain.log

[2017-09-01T10:38:34,940][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>401, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}}],\"type\":\"security_exception\",\"reason\":\"missing authentication token for REST request [/_bulk]\",\"header\":{\"WWW-Authenticate\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"}},\"status\":401}"}

我以为Kibana有安全认证，输入密码登录就好，没想到Logstash访问Elasticsearch也要认证。那我就老老实实在Logstash配置中添加认证信息

output {
  elasticsearch {
    hosts => ["http://10.10.40.170:9200"]
    user => logstash_system
    password => changeme
    manage_template => true
    index => "logstash-%{type}-%{+YYYY-MM-dd}"
  }

  stdout { codec => rubydebug }
}

主要是加入了账号和密码。
OK。重启开始验证。
。。。
还是不行，再次查看日志,发现每三秒出现一次如下的错误：

[2017-09-01T22:37:05,902][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_system:xxxxxx@localhost:9200/, :path=>"/"}
[2017-09-01T22:37:05,907][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_system:xxxxxx@localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :error=>"Elasticsearch Unreachable: [http://logstash_system:xxxxxx@localhost:9200/][Manticore::SocketException] Connection refused (Connection refused)"}
[2017-09-01T22:37:08,037][WARN ][logstash.shutdownwatcher ] {"inflight_count"=>1, "stalling_thread_info"=>{"other"=>[{"thread_id"=>24, "name"=>"[.monitoring-logstash]>worker0", "current_call"=>"[...]/vendor/bundle/jruby/1.9/gems/stud-0.0.23/lib/stud/interval.rb:89:in `sleep'"}]}}

我明明配置了Elasticsearch的hosts，怎么还是loccalhost。百思不得其解，陷入抓狂中。
一度以为是这个配置文件格式有问题，但是后来想想并不是，格式不对应该提示格式错误，这分明是网络错误。
最后还是仔细分析错误日志，health check是安装X-Pack之后才出现的，一查原来是X-Pack的监控。
https://www.elastic.co/guide/en/x-pack/current/monitoring-logstash.html
根据文中所说，这里默认的elasticsearch.url 就是那个该死的localhost:9200。

找到了问题根本所在，打开/etc/logstash/logstash.yml重新配置即可

xpack.monitoring.elasticsearch.url: ["http://10.10.40.170"] 
xpack.monitoring.elasticsearch.username: "logstash_system" 
xpack.monitoring.elasticsearch.password: "changeme"

重启之后，Logstash又恢复了正常。