HttpClient:Timeout waiting for connection from pool

最近线上服务经常抛出如下异常：

org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool

使用以下代码打出HttpClient连接池的状态：

Iterator iterator = poolingHttpClientConnectionManager.getRoutes().iterator();
while (iterator.hasNext()) {
    HttpRoute httpRoute = (HttpRoute) iterator.next();
    LOGGER.info(httpRoute.getTargetHost().getHostName() + poolingHttpClientConnectionManager.getStats(httpRoute));
}
LOGGER.info("---------------------totalStats:{}", poolingHttpClientConnectionManager.getTotalStats());

1.com[leased: 23; pending: 0; available: 2; max: 200]
2.com[leased: 0; pending: 0; available: 2; max: 200]
3.com[leased: 0; pending: 0; available: 2; max: 200]
---------------------totalStats:[leased: 23; pending: 0; available: 6; max: 600]

发现路由2.com和3.com处于正常状态，连接都能正常被release，但路由1.com的leased数量却只增不减，2、3与1有什么区别？
再看看tcp连接状态

# netstat -t | awk '/^tcp/ {++S[$NF]} END {for(a in S) print a, S[a]}'  
CLOSE_WAIT 27
ESTABLISHED 28
TIME_WAIT 4865

为何1.com的leased数量与CLOSE_WAIT数量如此接近？
看看代码，请求2.com和3.com用的是RestTemplate，而1.com因为当时遇到了url编码的问题，采用了如下方式：

try {
    BasicHttpRequest request = new BasicHttpRequest("GET", uri);
    HttpResponse response = httpClient.execute(TARGET, request);
    if (response.getStatusLine().getStatusCode() == HttpStatus.OK.value()) {
        String res = EntityUtils.toString(response.getEntity());
        Map result = JsonUtils.parse(res, Map.class);
        ...
    }
} catch (Exception e) {
    ...
} 

发起请求，响应码为200时处理，没问题！戳进源码看看

    // 原来 HttpResponse#getEntity() 有一步close
    // CloseableHttpResponseProxy#close()
    public void close() throws IOException {
        final HttpEntity entity = this.original.getEntity();
        EntityUtils.consume(entity);
    }
    // EntityUtils#consume()
    public static void consume(HttpEntity entity) throws IOException {
        if (entity != null) {
            if (entity.isStreaming()) {
                InputStream instream = entity.getContent();
                if (instream != null) {
                    instream.close();
                }
            }
        }
    }

原来HttpResponse#getEntity()时会释放资源，也就是将Entity消费掉！而上边代码中只有响应码为200时才会消费Entity，一旦响应码为非200时，资源没有关闭，连接得不到释放而转为CLOSE_WAIT状态，逐渐累积......
不敢想，解决问题先，把response.getEntity()代码提到响应码判断之前

HttpResponse response = httpClient.execute(TARGET, request);
String res = EntityUtils.toString(response.getEntity());
if (response.getStatusLine().getStatusCode() == HttpStatus.OK.value()) {
   ...
}

服务恢复正常！