KUDU同步数据到Hive报错

KUDU同步数据到Hive报错
一：现象
Azkaban每晚任务：将KUDU数据写入Hive
偶尔出现连接kudu报错：
Exception in thread “main” java.sql.SQLException: Unable to open scanner: Timed out: Client connection negotiation failed: client connection to 10.x.x.158:7050: BlockingWrite timed out: Timed out: Client connection negotiation failed: client connection to 10.x.x.168:7050
第一次出现时，重新执行任务后正常
但是过了一周后又出现该问题。于是入手排查（第一次出现就应该排查原因）
二：排查
1.怀疑是该ip的节点在同步时间段执行其他任务，资源不足导致
遂查看了cpu资源在01：00时在92%，确实有点高，但是不至于查不了Kudu
2.根据azkaban报错信息查看相对应两台服务器的kudu日志：
发现其中一台在同步时间点有如下日志信息

W0622 23:18:07.447243 14367 leader_election.cc:287] T f79d2020a3fc426787576141669a57f7 P 2b0320de8d0b4c14ab507ddd07d93a83 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 35222f8138fe4b1b9a7c49eb11287de0: Timed out: RequestConsensusVote RPC to 10.x.x.168:7050 timed out after 1.974s (SENT)
W0707 23:32:53.751653 14365 consensus_peers.cc:428] T 57a5267a51ca4eee9c74e2c1c35ddd95 P 2b0320de8d0b4c14ab507ddd07d93a83 -> Peer 732a535e5f96420fa8046e58d3305259 (h167:7050): Couldn't send request to peer 732a535e5f96420fa8046e58d3305259 for tablet 57a5267a51ca4eee9c74e2c1c35ddd95. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57a5267a51ca4eee9c74e2c1c35ddd95. Retrying in the next heartbeat period. Already tried 1 times.

看了下确实没有57a5267a51ca4eee9c74e2c1c35ddd95这个tablet，再看其他时间点，也有这些报错信息，说明并不是在同步时间才出现的问题。
再看日志发现：

W0708 11:36:44.159759 14366 consensus_peers.cc:428] T 57a5267a51ca4eee9c74e2c1c35ddd95 P 2b0320de8d0b4c14ab507ddd07d93a83 -> Peer f908fc5e8f534e9aa025d20dc7f54425 (h42:7050): Couldn't send request to peer f908fc5e8f534e9aa025d20dc7f54425 for tablet 57a5267a51ca4eee9c74e2c1c35ddd95. Status: Remote error: Service unavailable: Soft memory limit exceeded (at 100.03% of capacity). Retrying in the next heartbeat period. Already tried 102 times.

Soft memory limit exceeded (at 100.03% of capacity)
很明显是超出了内存限制。
查看了一下服务器的内存，还有40G，排除
对比了一下limit.conf，正常，排除
查看CDH-kudu内存设置：

memory.soft_limit_in_bytes是-1，说明没有限制。
调整Hard Memory Limit和Block Cache Capacity

三:重启Kudu
1.停止kudu写入业务（JStorm，SparkStreaming）
2.重启Kudu
3.等待所有的Tablet都打开block，端口7050打开，在kudu-ui界面的tablets中正常显示
4.启动业务（JStorm，SparkStreaming）