【kettle】pentaho/data-integration 报错：org.apache.http.NoHttpResponseException: failed to respond
一、版本说明
kettle版本：8.2.0.0-342
【kettle】pentaho/data-integration debug 查看日志方法
二、报错说明：
核心报错内容：
org.apache.http.NoHttpResponseException: xxx.com:80 failed to respond
执行到某一个请求时候报错，内容：
2024/01/18 15:24:06 - 获取json.0 - Connecting to [http://xxx.com/apis/query?id=123456] ...
2024/01/18 15:24:06 - 获取json.0 - Header parameter [Authorization]='Bearer ***'
15:24:06,404 DEBUG [BasicClientConnectionManager] Get connection for route {}->http://xxx.com:80
15:24:06,405 DEBUG [DefaultClientConnectionOperator] Connecting to xxx.com:80
15:24:06,407 DEBUG [RequestAddCookies] CookieSpec selected: default
15:24:06,407 DEBUG [RequestAuthCache] Auth cache not set in the context
15:24:06,407 DEBUG [RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
15:24:06,407 DEBUG [DefaultHttpClient] Attempt 1 to execute request
15:24:06,408 DEBUG [DefaultClientConnection] Sending request: POST /apis/query?id=123456 HTTP/1.1
15:24:06,408 DEBUG [wire] >> "POST /apis/query?id=123456 HTTP/1.1[\r][\n]"
15:24:06,408 DEBUG [wire] >> "Authorization: Bearer ***[\r][\n]"
15:24:06,408 DEBUG [wire] >> "Content-Type: application/json[\r][\n]"
15:24:06,408 DEBUG [wire] >> "Content-Length: 0[\r][\n]"
15:24:06,408 DEBUG [wire] >> "Host: xxx.com[\r][\n]"
15:24:06,408 DEBUG [wire] >> "Connection: Keep-Alive[\r][\n]"
15:24:06,408 DEBUG [wire] >> "User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)[\r][\n]"
15:24:06,408 DEBUG [wire] >> "[\r][\n]"
15:24:06,408 DEBUG [headers] >> POST /apis/query?id=123456 HTTP/1.1
15:24:06,408 DEBUG [headers] >> Authorization: Bearer ***
15:24:06,408 DEBUG [headers] >> Content-Type: application/json
15:24:06,408 DEBUG [headers] >> Content-Length: 0
15:24:06,408 DEBUG [headers] >> Host: xxx.com
15:24:06,408 DEBUG [headers] >> Connection: Keep-Alive
15:24:06,408 DEBUG [headers] >> User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)
15:24:06,661 DEBUG [DefaultClientConnection] Connection 0.0.0.0:63956<->10.64.252.13:80 closed
15:24:06,661 DEBUG [DefaultHttpClient] Closing the connection.
15:24:06,666 DEBUG [DefaultClientConnection] Connection 0.0.0.0:63956<->10.64.252.13:80 closed
15:24:06,666 DEBUG [DefaultClientConnection] Connection 0.0.0.0:63956<->10.64.252.13:80 shut down
15:24:06,666 DEBUG [BasicClientConnectionManager] Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@7adaefad
2024/01/18 15:24:06 - 获取json.0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : Because of an error, this step can't continue:
2024/01/18 15:24:06 - 获取json.0 - Can not result from [http://xxx.com/apis/query?id=123456]
2024/01/18 15:24:06 - 获取json.0 - org.apache.http.NoHttpResponseException: xxx.com:80 failed to respond
2024/01/18 15:24:06 - 获取json.0 - ERROR (version 8.2.0.0-342, build 8.2.0.0-342 from 2018-11-14 10.30.55 by buildguy) : org.pentaho.di.core.exception.KettleException:
2024/01/18 15:24:06 - 获取json.0 - Can not result from [http://xxx.com/apis/query?id=123456]
2024/01/18 15:24:06 - 获取json.0 - org.apache.http.NoHttpResponseException: xxx.com:80 failed to respond
2024/01/18 15:24:06 - 获取json.0 -
2024/01/18 15:24:06 - 获取json.0 - at org.pentaho.di.trans.steps.rest.Rest.callRest(Rest.java:273)
2024/01/18 15:24:06 - 获取json.0 - at org.pentaho.di.trans.steps.rest.Rest.processRow(Rest.java:470)
2024/01/18 15:24:06 - 获取json.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2024/01/18 15:24:06 - 获取json.0 - at java.lang.Thread.run(Thread.java:748)
2024/01/18 15:24:06 - 获取json.0 - Caused by: com.sun.jersey.api.client.ClientHandlerException: org.apache.http.NoHttpResponseException: xxx.com:80 failed to respond
2024/01/18 15:24:06 - 获取json.0 - at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
2024/01/18 15:24:06 - 获取json.0 - at com.sun.jersey.api.client.Client.handle(Client.java:652)
2024/01/18 15:24:06 - 获取json.0 - at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
2024/01/18 15:24:06 - 获取json.0 - at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
2024/01/18 15:24:06 - 获取json.0 - at com.sun.jersey.api.client.WebResource$Builder.post(WebResource.java:570)
2024/01/18 15:24:06 - 获取json.0 - at org.pentaho.di.trans.steps.rest.Rest.callRest(Rest.java:188)
2024/01/18 15:24:06 - 获取json.0 - ... 3 more
2024/01/18 15:24:06 - 获取json.0 - Caused by: org.apache.http.NoHttpResponseException: xxx.com:80 failed to respond
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:281)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:207)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:684)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
2024/01/18 15:24:06 - 获取json.0 - at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
2024/01/18 15:24:06 - 获取json.0 - at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
2024/01/18 15:24:06 - 获取json.0 - ... 8 more
对比看下不报错查询的日志：
15:24:06,225 DEBUG [BasicClientConnectionManager] Get connection for route {}->http://xxx.com:80
15:24:06,225 DEBUG [DefaultClientConnectionOperator] Connecting to xxx.com:80
15:24:06,228 DEBUG [RequestAddCookies] CookieSpec selected: default
15:24:06,228 DEBUG [RequestAuthCache] Auth cache not set in the context
15:24:06,228 DEBUG [RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
15:24:06,228 DEBUG [DefaultHttpClient] Attempt 1 to execute request
15:24:06,228 DEBUG [DefaultClientConnection] Sending request: POST /apis/query?id=123456 HTTP/1.1
15:24:06,228 DEBUG [wire] >> "POST /apis/query?id=123456 HTTP/1.1[\r][\n]"
15:24:06,228 DEBUG [wire] >> "Authorization: Bearer ***[\r][\n]"
15:24:06,228 DEBUG [wire] >> "Content-Type: application/json[\r][\n]"
15:24:06,228 DEBUG [wire] >> "Content-Length: 0[\r][\n]"
15:24:06,228 DEBUG [wire] >> "Host: xxx.com[\r][\n]"
15:24:06,228 DEBUG [wire] >> "Connection: Keep-Alive[\r][\n]"
15:24:06,228 DEBUG [wire] >> "User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)[\r][\n]"
15:24:06,228 DEBUG [wire] >> "[\r][\n]"
15:24:06,228 DEBUG [headers] >> POST /apis/query?id=123456 HTTP/1.1
15:24:06,228 DEBUG [headers] >> Authorization: Bearer ***
15:24:06,228 DEBUG [headers] >> Content-Type: application/json
15:24:06,228 DEBUG [headers] >> Content-Length: 0
15:24:06,228 DEBUG [headers] >> Host: xxx.com
15:24:06,229 DEBUG [headers] >> Connection: Keep-Alive
15:24:06,229 DEBUG [headers] >> User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)
15:24:06,343 DEBUG [wire] << "HTTP/1.1 200 [\r][\n]"
15:24:06,343 DEBUG [wire] << "Content-Type: application/json[\r][\n]"
15:24:06,347 DEBUG [wire] << "Transfer-Encoding: chunked[\r][\n]"
15:24:06,347 DEBUG [wire] << "Connection: keep-alive[\r][\n]"
15:24:06,347 DEBUG [wire] << "Vary: Origin[\r][\n]"
15:24:06,347 DEBUG [wire] << "X-Content-Type-Options: nosniff[\r][\n]"
15:24:06,347 DEBUG [wire] << "X-XSS-Protection: 1; mode=block[\r][\n]"
15:24:06,347 DEBUG [wire] << "Cache-Control: no-cache, no-store, max-age=0, must-revalidate[\r][\n]"
15:24:06,347 DEBUG [wire] << "Pragma: no-cache[\r][\n]"
15:24:06,347 DEBUG [wire] << "Expires: 0[\r][\n]"
15:24:06,347 DEBUG [wire] << "Date: Thu, 18 Jan 2024 07:24:06 GMT[\r][\n]"
15:24:06,347 DEBUG [wire] << "Access-Control-Allow-Origin: *[\r][\n]"
15:24:06,347 DEBUG [wire] << "X-Kong-Upstream-Latency: 113[\r][\n]"
15:24:06,347 DEBUG [wire] << "X-Kong-Proxy-Latency: 0[\r][\n]"
15:24:06,348 DEBUG [wire] << "Via: kong/2.7.0[\r][\n]"
15:24:06,348 DEBUG [wire] << "vary: Origin[\r][\n]"
15:24:06,348 DEBUG [wire] << "[\r][\n]"
15:24:06,348 DEBUG [DefaultClientConnection] Receiving response: HTTP/1.1 200
15:24:06,348 DEBUG [headers] << HTTP/1.1 200
15:24:06,348 DEBUG [headers] << Content-Type: application/json
15:24:06,348 DEBUG [headers] << Transfer-Encoding: chunked
15:24:06,348 DEBUG [headers] << Connection: keep-alive
15:24:06,348 DEBUG [headers] << Vary: Origin
15:24:06,348 DEBUG [headers] << X-Content-Type-Options: nosniff
15:24:06,348 DEBUG [headers] << X-XSS-Protection: 1; mode=block
15:24:06,348 DEBUG [headers] << Cache-Control: no-cache, no-store, max-age=0, must-revalidate
15:24:06,348 DEBUG [headers] << Pragma: no-cache
15:24:06,348 DEBUG [headers] << Expires: 0
15:24:06,348 DEBUG [headers] << Date: Thu, 18 Jan 2024 07:24:06 GMT
15:24:06,348 DEBUG [headers] << Access-Control-Allow-Origin: *
15:24:06,348 DEBUG [headers] << X-Kong-Upstream-Latency: 113
15:24:06,348 DEBUG [headers] << X-Kong-Proxy-Latency: 0
15:24:06,348 DEBUG [headers] << Via: kong/2.7.0
15:24:06,348 DEBUG [headers] << vary: Origin
15:24:06,348 DEBUG [DefaultHttpClient] Connection can be kept alive indefinitely
15:24:06,349 DEBUG [wire] << "38[\r][\n]"
15:24:06,349 DEBUG [wire] << "隐藏内容}"
2024/01/18 15:24:06 - 获取json.0 - Response time (milliseconds): [125] for [http://xxx.com/apis/query?id=123456]
2024/01/18 15:24:06 - 获取json.0 - The response code is 200
15:24:06,349 DEBUG [wire] << "[\r][\n]"
15:24:06,349 DEBUG [wire] << "0[\r][\n]"
15:24:06,349 DEBUG [wire] << "[\r][\n]"
15:24:06,349 DEBUG [BasicClientConnectionManager] Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@4bdd086a
15:24:06,349 DEBUG [BasicClientConnectionManager] Connection can be kept alive indefinitely
三、报错原因分析
参考文章 记一次NoHttpResponseException：xxx failed to respond
得知kettle使用的是 apache的httpclient作为
三个组件的连接工具。
而问题原因归结为：keep-alive配置
于是手动搭建springboot项目并增加配置：
定制KeepAliveTimeout，设置10秒；5个请求则自动断开keepalive连接
import org.apache.catalina.connector.Connector;
import org.apache.coyote.http11.Http11NioProtocol;
import org.springframework.boot.web.embedded.tomcat.TomcatConnectorCustomizer;
import org.springframework.boot.web.embedded.tomcat.TomcatServletWebServerFactory;
import org.springframework.boot.web.server.ConfigurableWebServerFactory;
import org.springframework.boot.web.server.WebServerFactoryCustomizer;
import org.springframework.context.annotation.Configuration;
@Configuration
public class WebServerConfiguration implements WebServerFactoryCustomizer<ConfigurableWebServerFactory> {
@Override
public void customize(ConfigurableWebServerFactory factory) {
//使用对应工厂类提供给我们的接口定制化我们的tomcat connector
((TomcatServletWebServerFactory) factory).addConnectorCustomizers(new TomcatConnectorCustomizer() {
@Override
public void customize(Connector connector) {
Http11NioProtocol protocol = (Http11NioProtocol) connector.getProtocolHandler();
//定制KeepAliveTimeout，设置10秒内没有请求则服务器自动断开keepalive连接
protocol.setKeepAliveTimeout(10000);
//当客户端发送超过5个请求则自动断开keepalive连接
protocol.setMaxKeepAliveRequests(5);
}
});
}
}
增加测试类：
关闭重试，强制keepAlive=-1
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpPost;
import org.apache.http.client.utils.URIBuilder;
import org.apache.http.conn.ConnectionKeepAliveStrategy;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy;
import org.apache.http.impl.client.DefaultHttpRequestRetryHandler;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.impl.conn.PoolingHttpClientConnectionManager;
import org.apache.http.util.EntityUtils;
import org.junit.jupiter.api.Test;
import java.io.IOException;
import java.net.URI;
import java.net.URISyntaxException;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Map;
public class HttpRequest {
// 默认keepalive策略 ，会获取response中的keepalive参数并配置到client中
private ConnectionKeepAliveStrategy udfKeepAliveStrategy = DefaultConnectionKeepAliveStrategy.INSTANCE;
// 强制 -1表示无论如何都任务server端不会关闭连接
private ConnectionKeepAliveStrategy noneKeepAliveStrategy = (response, context) -> -1;
// 如果不配置此项，也会有添加默认配置重试3次，此处增加重试次数。
private DefaultHttpRequestRetryHandler udfRetryHandler = new DefaultHttpRequestRetryHandler(30, false);
private PoolingHttpClientConnectionManager manager;
private String INCR_URL = "http://localhost:8080/api/v1/incr";
public HttpRequest() {
manager = new PoolingHttpClientConnectionManager();
manager.setDefaultMaxPerRoute(100);
manager.setMaxTotal(200);
manager.setValidateAfterInactivity(10_000);
}
public CloseableHttpClient getClient() {
HttpClientBuilder httpClientBuilder = HttpClientBuilder.create();
httpClientBuilder
.setConnectionManager(manager)
// .setRetryHandler(udfRetryHandler)
// .setKeepAliveStrategy(udfKeepAliveStrategy)
.setKeepAliveStrategy(noneKeepAliveStrategy)
.disableAutomaticRetries()
;
CloseableHttpClient client = httpClientBuilder.build();
return client;
}
@Test
public void httpRequest() throws URISyntaxException, IOException {
CloseableHttpClient client = getClient();
URI uri = new URIBuilder(INCR_URL + "/info").build();
for (int i = 0; i < 10; i++) {
HttpPost post = new HttpPost(uri);
CloseableHttpResponse response = client.execute(post);
Map<String, String> headerMap = new HashMap<>();
Arrays.stream(response.getAllHeaders()).forEach(f->headerMap.put(f.getName(),f.getValue()));
String responseStr = EntityUtils.toString(response.getEntity());
String headersStr=headerMap.toString();
System.out.println(String.format("content: %s, headers: %s",responseStr,headersStr));
}
}
}
经过测试：
16:01:57.448 [main] DEBUG org.apache.http.client.protocol.RequestAddCookies - CookieSpec selected: default
16:01:57.448 [main] DEBUG org.apache.http.client.protocol.RequestAuthCache - Auth cache not set in the context
16:01:57.448 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection request: [route: {}->http://localhost:8080][total kept alive: 1; route allocated: 1 of 100; total allocated: 1 of 200]
16:01:57.448 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection leased: [id: 1][route: {}->http://localhost:8080][total kept alive: 0; route allocated: 1 of 100; total allocated: 1 of 200]
16:01:57.448 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request POST /api/v1/incr/info HTTP/1.1
16:01:57.448 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED
16:01:57.448 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED
16:01:57.448 [main] DEBUG org.apache.http.headers - http-outgoing-1 >> POST /api/v1/incr/info HTTP/1.1
16:01:57.448 [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Content-Length: 0
16:01:57.448 [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Host: localhost:8080
16:01:57.448 [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Connection: Keep-Alive
16:01:57.448 [main] DEBUG org.apache.http.headers - http-outgoing-1 >> User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)
16:01:57.448 [main] DEBUG org.apache.http.headers - http-outgoing-1 >> Accept-Encoding: gzip,deflate
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "POST /api/v1/incr/info HTTP/1.1[\r][\n]"
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Content-Length: 0[\r][\n]"
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Host: localhost:8080[\r][\n]"
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]"
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)[\r][\n]"
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "Accept-Encoding: gzip,deflate[\r][\n]"
16:01:57.448 [main] DEBUG org.apache.http.wire - http-outgoing-1 >> "[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "HTTP/1.1 200 [\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Content-Type: application/json[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Transfer-Encoding: chunked[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Date: Wed, 17 Jan 2024 08:01:57 GMT[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "Connection: close[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "18[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "{"id":9,"name":"info-9"}[\r][\n]"
16:01:57.758 [main] DEBUG org.apache.http.headers - http-outgoing-1 << HTTP/1.1 200
16:01:57.758 [main] DEBUG org.apache.http.headers - http-outgoing-1 << Content-Type: application/json
16:01:57.758 [main] DEBUG org.apache.http.headers - http-outgoing-1 << Transfer-Encoding: chunked
16:01:57.758 [main] DEBUG org.apache.http.headers - http-outgoing-1 << Date: Wed, 17 Jan 2024 08:01:57 GMT
16:01:57.758 [main] DEBUG org.apache.http.headers - http-outgoing-1 << Connection: close
16:01:57.759 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "0[\r][\n]"
16:01:57.759 [main] DEBUG org.apache.http.wire - http-outgoing-1 << "[\r][\n]"
16:01:57.759 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection - http-outgoing-1: Close connection
16:01:57.759 [main] DEBUG org.apache.http.impl.execchain.MainClientExec - Connection discarded
16:01:57.759 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 1][route: {}->http://localhost:8080][total kept alive: 0; route allocated: 0 of 100; total allocated: 0 of 200]
结果：
程序根本不会报错！！！但是debug srpingboot端设置的close connection生效了，处理每5条日志此日志就会打印获取的response中含有Connection: close内容。apache 会根据此内容，重建connection。所以没有任何报错内容。备注：其他记录都含有Connection: keep-alive字样，表示服务器还没关闭connection。
四、解决方法
办法就是增加重试配置。
4.1 准备工作：
把kettle/lib中的包install到本地仓库：
需要的kettle包都在kettle安装目录/lib下：KETTLE_HOME/lib
mvn install:install-file "-DgroupId=pentaho-kettle" "-DartifactId=kettle-core" "-Dversion=8.2.0.0-342" "-Dpackaging=jar" "-Dfile=install/kettle-core-8.2.0.0-342.jar"
mvn install:install-file "-DgroupId=pentaho-kettle" "-DartifactId=kettle-engine" "-Dversion=8.2.0.0-342" "-Dpackaging=jar" "-Dfile=install/kettle-engine-8.2.0.0-342.jar"
mvn install:install-file "-DgroupId=pentaho" "-DartifactId=metastore" "-Dversion=8.2.0.0-342" "-Dpackaging=jar" "-Dfile=install/metastore-8.2.0.0-342.jar"
新建maven项目：
pom.xml增加依赖：
<dependency>
<groupId>pentaho-kettlegroupId>
<artifactId>kettle-coreartifactId>
<version>8.2.0.0-342version>
<scope>providedscope>
dependency>
<dependency>
<groupId>pentaho-kettlegroupId>
<artifactId>kettle-engineartifactId>
<version>8.2.0.0-342version>
<scope>providedscope>
dependency>
<dependency>
<groupId>pentahogroupId>
<artifactId>metastoreartifactId>
<version>8.2.0.0-342version>
<scope>providedscope>
dependency>
<dependency>
<groupId>org.apache.httpcomponentsgroupId>
<artifactId>httpclientartifactId>
<version>4.5.3version>
<scope>providedscope>
dependency>
<dependency>
<groupId>commons-langgroupId>
<artifactId>commons-langartifactId>
<version>2.6version>
<scope>providedscope>
dependency>
<dependency>
<groupId>com.googlecode.json-simplegroupId>
<artifactId>json-simpleartifactId>
<version>1.1version>
<scope>providedscope>
dependency>
<dependency>
<groupId>com.github.rholdergroupId>
<artifactId>guava-retryingartifactId>
<version>2.0.0version>
<scope>providedscope>
dependency>
4.2 解决方法一：
问题解决
所以按照之前参考文章的内容： org.pentaho.di.cluster.SlaveConnectionManager 和 org.pentaho.di.core.util.HttpClientManager 类中
修改所有使用 HttpClients 和 HttpClientBuilder 新建 client的位置，增加retry相关配置
注意：使用DefaultHttpRequestRetryHandler必须设置true，或者使用StandardHttpRequestRetryHandler也可以。
// private ConnectionKeepAliveStrategy udfKeepAliveStrategy = DefaultConnectionKeepAliveStrategy.INSTANCE;
// private DefaultClientConnectionReuseStrategy udfReuseHandler= new DefaultClientConnectionReuseStrategy();
private HttpRequestRetryHandler udfRetryHandler=new DefaultHttpRequestRetryHandler(5,true); // 必须true!
// private HttpRequestRetryHandler udfRetryHandler = new StandardHttpRequestRetryHandler(5,true) // true false无所谓
KeepAlive和ReuseStrategy默认都是会添加的，所以不配置也可以。retry策略必须添加。
编译，用刚刚编译的替换kettle-core-8.2.0.0-342.jar中的.class文件，并替换KETTLE_HOME/lib/kettle-core-8.2.0.0-342.jar包。
用kettle调用自己的写的springboot服务看下：
使用kettle的rest组件调用自己的springboot接口发现，cmd窗口打印的debug日志response信息中不会出现：Connection: close。所以kettle中rest或者http在每条数据运行过程都是新建了一个apache httpclient对象，就更没有复用connection！每条数据服务器返回的response都有Connection: keep-alive，看来效率很低。
2024/01/17 13:35:04 - 写日志.0 -
2024/01/17 13:35:04 - 写日志.0 - ------------> 行号 99------------------------------
2024/01/17 13:35:04 - 写日志.0 - res = {"id":1510,"name":"info-1510"}
2024/01/17 13:35:04 - 写日志.0 -
2024/01/17 13:35:04 - 写日志.0 - ====================
2024/01/17 13:35:04 - REST client.0 - Connecting to [http://localhost:8080/api/v1/incr/info] ...
2024/01/17 13:35:04 - REST client.0 - Connecting to [http://localhost:8080/api/v1/incr/info] ...
2024/01/17 13:35:04 - REST client.0 - Adding HTTP body value [1]
13:35:04,357 DEBUG [BasicClientConnectionManager] Get connection for route {}->http://localhost:8080
13:35:04,358 DEBUG [DefaultClientConnectionOperator] Connecting to localhost:8080
13:35:04,359 DEBUG [RequestAddCookies] CookieSpec selected: default
13:35:04,359 DEBUG [RequestAuthCache] Auth cache not set in the context
13:35:04,359 DEBUG [RequestTargetAuthentication] Target auth state: UNCHALLENGED
13:35:04,359 DEBUG [RequestProxyAuthentication] Proxy auth state: UNCHALLENGED
13:35:04,359 DEBUG [DefaultHttpClient] Attempt 1 to execute request
13:35:04,359 DEBUG [DefaultClientConnection] Sending request: POST /api/v1/incr/info HTTP/1.1
13:35:04,359 DEBUG [wire] >> "POST /api/v1/incr/info HTTP/1.1[\r][\n]"
13:35:04,359 DEBUG [wire] >> "Content-Type: application/json[\r][\n]"
13:35:04,359 DEBUG [wire] >> "Transfer-Encoding: chunked[\r][\n]"
13:35:04,359 DEBUG [wire] >> "Host: localhost:8080[\r][\n]"
13:35:04,359 DEBUG [wire] >> "Connection: Keep-Alive[\r][\n]"
13:35:04,359 DEBUG [wire] >> "User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)[\r][\n]"
13:35:04,360 DEBUG [wire] >> "[\r][\n]"
13:35:04,360 DEBUG [headers] >> POST /api/v1/incr/info HTTP/1.1
13:35:04,360 DEBUG [headers] >> Content-Type: application/json
13:35:04,360 DEBUG [headers] >> Transfer-Encoding: chunked
13:35:04,360 DEBUG [headers] >> Host: localhost:8080
13:35:04,360 DEBUG [headers] >> Connection: Keep-Alive
13:35:04,360 DEBUG [headers] >> User-Agent: Apache-HttpClient/4.5.3 (Java/1.8.0_261)
13:35:04,360 DEBUG [wire] >> "1[\r][\n]"
13:35:04,360 DEBUG [wire] >> "1"
13:35:04,360 DEBUG [wire] >> "[\r][\n]"
13:35:04,360 DEBUG [wire] >> "0[\r][\n]"
13:35:04,360 DEBUG [wire] >> "[\r][\n]"
13:35:04,674 DEBUG [wire] << "HTTP/1.1 200 [\r][\n]"
13:35:04,675 DEBUG [wire] << "Content-Type: application/json[\r][\n]"
13:35:04,680 DEBUG [wire] << "Transfer-Encoding: chunked[\r][\n]"
13:35:04,680 DEBUG [wire] << "Date: Wed, 17 Jan 2024 05:35:04 GMT[\r][\n]"
13:35:04,680 DEBUG [wire] << "Keep-Alive: timeout=10[\r][\n]"
13:35:04,680 DEBUG [wire] << "Connection: keep-alive[\r][\n]"
13:35:04,680 DEBUG [wire] << "[\r][\n]"
13:35:04,680 DEBUG [DefaultClientConnection] Receiving response: HTTP/1.1 200
13:35:04,680 DEBUG [headers] << HTTP/1.1 200
13:35:04,680 DEBUG [headers] << Content-Type: application/json
13:35:04,680 DEBUG [headers] << Transfer-Encoding: chunked
13:35:04,680 DEBUG [headers] << Date: Wed, 17 Jan 2024 05:35:04 GMT
13:35:04,680 DEBUG [headers] << Keep-Alive: timeout=10
13:35:04,680 DEBUG [headers] << Connection: keep-alive
13:35:04,681 DEBUG [DefaultHttpClient] Connection can be kept alive for 10000 MILLISECONDS
13:35:04,681 DEBUG [wire] << "1e[\r][\n]"
13:35:04,681 DEBUG [wire] << "{"id":1511,"name":"info-1511"}"
2024/01/17 13:35:04 - REST client.0 - Response time (milliseconds): [325] for [http://localhost:8080/api/v1/incr/info]
2024/01/17 13:35:04 - REST client.0 - The response code is 200
13:35:04,681 DEBUG [wire] << "[\r][\n]"
13:35:04,681 DEBUG [wire] << "0[\r][\n]"
13:35:04,682 DEBUG [wire] << "[\r][\n]"
13:35:04,682 DEBUG [BasicClientConnectionManager] Releasing connection org.apache.http.impl.conn.ManagedClientConnectionImpl@2192ee11
13:35:04,682 DEBUG [BasicClientConnectionManager] Connection can be kept alive for 10000 MILLISECONDS
现在可以断定不是keepalive配置或connection复用的问题了。
再次运行kettle访问生产服务，由于增加了retry，也没有报错了！
推测是kettle中的apache httpclient创建connection后进行get/post发现服务端connection就已经关闭了，是服务端问题，但服务端问题没权限解决。
apache的httpclient重试机制浅析
具体可见org.apache.http.impl.client.HttpClientBuilder
如果没有HttpClientBuilder没有配置retry则会在build是时候设置默认retryhandler
org.apache.http.impl.execchain.RetryExec就是重试的执行器，retryHandler.retryRequet的if判断最重要，重试过程如下：
但是设置了默认的org.apache.http.impl.client.DefaultHttpRequestRetryHandler重试handler是不是就可以重试呢，不是的，如下图：默认重试handler，要重试需要满足：
在限定次数3次内，requestSentRetryEnabled=ture或Request是幂等的。
如果使用默认的重试器，发起HttpPut或者HttpPost从源码推定不是幂等的，所以默认重试器不会重试。
StandardHttpRequestRetryHandler类中重写了handleAsIdempotent幂等判断的方法，基本所有请求都视为幂等得了，所以requestSentRetryEnabled配置成true/false就无所谓了。
ReuseStrategy和KeepAliveStrategy工作原理：
如下图从response获取的header信息：
org.apache.http.impl.client.DefaultClientConnectionReuseStrategy类的功能就是根据response返回值获取connection参数是否close并返回boolean值。
org.apache.http.impl.client.DefaultConnectionKeepAliveStrategy根据response解析connection_keep_alive的timeout值
reuse和keepalive的使用在主执行器中：
org.apache.http.impl.execchain.MainClientExec#execute方法，描述了reuse 和 keepalive 使用。
即，如果从response中获取connection=keep-alive则，再去获取keep-alive的timeout值并将此值回填至connection，待connection pool使用此connection前校验是否有效。
在org.apache.http.impl.client.HttpClientBuilder#build方法中都是默认添加reuse和keepalive的handler的。
4.3 解决方法二
可以在外层自己实现重试！
kettle中http或rest组件就是使用如下两个类：
org.pentaho.di.trans.steps.http.HTTP和org.pentaho.di.trans.steps.httppost.HTTPPOST
其中HTTP负责除POST外的请求，HTTPPOST负责POST。
这两个类都在kettle-engine-8.2.0.0-342.jar包中
把kettle包中的 org.pentaho.di.trans.steps.http.HTTP和org.pentaho.di.trans.steps.httppost.HTTPPOST 复制源码到自己的项目中：
新建一个retryutils，这里使用guava-retrying实现，也可以使用spring-retry，策略更丰富些。
此处重试次数和延迟时间都写死了，根据自己需求修改。
import com.github.rholder.retry.*;
import org.apache.http.NoHttpResponseException;
import org.apache.http.client.methods.CloseableHttpResponse;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.TimeUnit;
public class RetryUtils {
public static Retryer<CloseableHttpResponse> getHttpResponseRetryer() {
Retryer<CloseableHttpResponse> retryer = RetryerBuilder.<CloseableHttpResponse>newBuilder()
.retryIfExceptionOfType(NoHttpResponseException.class) //设置异常重试源
.retryIfResult(res -> res == null) //设置根据结果重试
.withWaitStrategy(WaitStrategies.fixedWait(2, TimeUnit.SECONDS)) //设置等待间隔时间
.withStopStrategy(StopStrategies.stopAfterAttempt(999)) //设置最大重试次数
.build();
return retryer;
}
public static CloseableHttpResponse getResponseWithRetry(Callable<CloseableHttpResponse> supplier) throws ExecutionException, RetryException {
Retryer<CloseableHttpResponse> retryer = getHttpResponseRetryer();
CloseableHttpResponse res = retryer.call(supplier);
return res;
}
}
修改源码：
把 org.pentaho.di.trans.steps.http.HTTP和org.pentaho.di.trans.steps.httppost.HTTPPOST 中的httpClient.execute都用RetryUtils.getResponseWithRetry给包起来，如下：
编译，并把原始kettle-engine-8.2.0.0-342.jar包中的HTTP.class和HTTPPOST.class用自己刚刚编译的替换掉。再把KETTLE_HOME/lib/kettle-engine-8.2.0.0-342.jar包替换掉。把guava-retrying-2.0.0.jar 也复制到KETTLE_HOME/lib下。
大功告成，启动kettle！
注意事项：
重新发起rest/http只是适用于请求数据操作，如果发起请求后的操作不是幂等的，重试机制就会造成服务端操作被执行多次，切记！！！