Jeager+cat+ELK 快速构建分布式调用链跟踪系统

1.组件介绍

	因为Jaeger 目前使用大公司较少，文档也比较少所以整理出来以供大家参考，如有更好的方式欢迎交流，这里主要以追踪Java的链路为介绍对象。
	CAT 由大众点评开发的，基于 Java 的实时应用监控平台，包括实时应用监控，业务监控。
	Jaeger是Uber推出的一款调用链追踪系统，类似于Zipkin和Dapper，为微服务调用追踪而生。 其主要用于多个服务调用过程追踪分析，图形化服务调用轨迹，便于快速准确定位问题。
	ELK 成熟的日志系统，这里采用filebaet---->logstash---->es---->kibana的方式。

2.架构图

3.部署方式
Cat:https://gitee.com/meijinmeng/doc.git 及注意修改配置
ELK:部署不做赘述，主要用filebeat采集log日志，上报到logstash，logstash在发送到es中。
这里主要介绍Jaeger部署几种常见的部署方式。

4.Jaeger 几种常见的常见的部署方式

**1):快速部署demo ---用内存存储数据 【重启后数据会消失】**
docker run -d --name jaeger   -e COLLECTOR_ZIPKIN_HTTP_PORT=9411   -p 5775:5775/udp   -p 6831:6831/udp   -p 6832:6832/udp   -p 5778:5778   -p 16686:16686   -p 14268:14268   -p 14250:14250   -p 9411:9411   jaegertracing/all-in-one:1.18


**2):集成本地存储一键部署,挂载本地存储，不用集成ES 【重启后数据不会丢失】**
docker run -d --name jaeger --restart=always \
  -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \
  -e SPAN_STORAGE_TYPE=badger \
  -e BADGER_EPHEMERAL=false \
  -e BADGER_DIRECTORY_VALUE=/badger/data \
  -e BADGER_DIRECTORY_KEY=/badger/key \
  -v /badger:/badger \ #mdkri /badger 挂在本地后数据不会丢失。
  -p 5775:5775/udp \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:1.18


**3):ES+Jaeger** 

**elasticsearch 部署**
	源码安装或者docker 安装都可以。
	192.168.1.48:9200

**collector 部署**

docker run -d --name jaeger-collector  --restart=always -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://192.168.1.48:9200 -e ES_USERNAME=elastic  -p 14250:14250  -p 14267:14267   -p 14268:14268    -p 9411:9411 jaegertracing/jaeger-collector



**query 部署** 
docker run -d --name jaeger-query --restart=always -e SPAN_STORAGE_TYPE=elasticsearch -e ES_SERVER_URLS=http://192.168.1.48:9200 -e ES_USERNAME=elastic  -e JAEGER_AGENT_HOST=172.17.0.4 -e AEGER_AGENT_PORT=6831  -p 16686:16686/tcp jaegertracing/jaeger-query


	{"level":"error","ts":1592364318.7607894,"caller":"zap/logger.go:33","msg":"failed to flush Jaeger spans to server: write udp 127.0.0.1:35361->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\tgithub.com/uber/[email protected]+incompatible/log/zap/logger.go:33\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\tgithub.com/uber/[email protected]+incompatible/reporter.go:287\ngithub.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\tgithub.com/uber/[email protected]+incompatible/reporter.go:297"}
	解决：添加参数  -e JAEGER_AGENT_HOST=172.17.0.4 -e AEGER_AGENT_PORT=6831


**3.agent部署**
docker run   -d  --name jaeger-agent --restart=always  -p 5775:5775/udp   -p 6831:6831/udp   -p 6832:6832/udp   -p 5778:5778/tcp   jaegertracing/jaeger-agent   --reporter.grpc.host-port=192.168.1.201:14250



		{"level":"warn","ts":1592373273.73611,"caller":"[email protected]/clientconn.go:1223","msg":"grpc: addrConn.createTransport failed to connect to {192.168.1.228:14267   0 }. Err :write tcp 172.17.0.4:42142->192.168.1.228:14267: write: broken pipe. Reconnecting...","system":"grpc","grpc_log":true}
		解决：更换collector -p端口映射顺序

		{"level":"warn","ts":1592449922.2165968,"caller":"[email protected]/clientconn.go:1223","msg":"grpc: addrConn.createTransport failed to connect to {http://172.17.0.2:14268   0 }. Err :connection error: desc = \"transport: Error while dialing dial tcp: address http://172.17.0.2:14268: too many colons in address\". Reconnecting...","system":"grpc","grpc_log":true}

		{"level":"error","ts":1592447099.2771292,"caller":"grpc/reporter.go:74","msg":"Could not send spans over gRPC","error":"rpc error: code = Unavailable desc = last connection error: connection closed","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(*Reporter).send\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:74\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(*Reporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:53\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(*MetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/metrics.go:83\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(*ClientMetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/client_metrics.go:120\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:138\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:112\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:121\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:86"}

		{"level":"error","ts":1592450114.5120075,"caller":"grpc/reporter.go:74","msg":"Could not send spans over gRPC","error":"rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing dial tcp: address http://172.17.0.2:14267: too many colons in address\"","stacktrace":"github.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(*Reporter).send\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:74\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc.(*Reporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/grpc/reporter.go:53\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(*MetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/metrics.go:83\ngithub.com/jaegertracing/jaeger/cmd/agent/app/reporter.(*ClientMetricsReporter).EmitBatch\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/reporter/client_metrics.go:120\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(*agentProcessorEmitBatch).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:138\ngithub.com/jaegertracing/jaeger/thrift-gen/jaeger.(*AgentProcessor).Process\n\tgithub.com/jaegertracing/jaeger/thrift-gen/jaeger/agent.go:112\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.(*ThriftProcessor).processBuffer\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:121\ngithub.com/jaegertracing/jaeger/cmd/agent/app/processors.NewThriftProcessor.func2\n\tgithub.com/jaegertracing/jaeger/cmd/agent/app/processors/thrift_processor.go:86"}
		
		*#解决：增加collector 端口号14250，agent配置--reporter.grpc.host-port=192.168.1.201:14250*

5.效果图