解决：FLINK1.12.0 Upsert Kafka SQL Connector 无法识别scan.startup.mode的问题

问题描述：

在升级Flink 1.12.0 以后 Upsert Kafka SQL Connector 不支持scan.startup.mode、scan.startup.specific-offsets、scan.startup.timestamp-millis
无法指定消费点的问题：

---

原因分析：

由于升级Flink 1.12.0以后Upsert kafka 取消了这些参数，并在社区提问云邪后，得到的回复是

---

解决方案：

经过对源码分析在KafkaDynamicTableFactory入口类里的optionalOptions方法里的语义参数有这些

@Override
	public Set<ConfigOption<?>> optionalOptions() {
		final Set<ConfigOption<?>> options = new HashSet<>();
		options.add(FactoryUtil.FORMAT);
		options.add(KEY_FORMAT);
		options.add(KEY_FIELDS);
		options.add(KEY_FIELDS_PREFIX);
		options.add(VALUE_FORMAT);
		options.add(VALUE_FIELDS_INCLUDE);
		options.add(TOPIC);
		options.add(TOPIC_PATTERN);
		options.add(PROPS_GROUP_ID);
		options.add(SCAN_STARTUP_MODE);
		options.add(SCAN_STARTUP_SPECIFIC_OFFSETS);
		options.add(SCAN_TOPIC_PARTITION_DISCOVERY);
		options.add(SCAN_STARTUP_TIMESTAMP_MILLIS);
		options.add(SINK_PARTITIONER);
		options.add(SINK_SEMANTIC);
		options.add(SINK_PARALLELISM);
		return options;
	}

而在UpsertKafkaDynamicTableFactory中却只有

@Override
	public Set<ConfigOption<?>> optionalOptions() {
		final Set<ConfigOption<?>> options = new HashSet<>();
		options.add(KEY_FIELDS_PREFIX);
		options.add(VALUE_FIELDS_INCLUDE);
		options.add(FactoryUtil.SINK_PARALLELISM);
		
		return options;
	}

再看KafkaDynamicTableFactory中创建kaTableSource的参数是：

return createKafkaTableSource(
			physicalDataType,
			keyDecodingFormat.orElse(null),
			valueDecodingFormat,
			keyProjection,
			valueProjection,
			keyPrefix,
			KafkaOptions.getSourceTopics(tableOptions),
			KafkaOptions.getSourceTopicPattern(tableOptions),
			properties,
			startupOptions.startupMode,
			startupOptions.specificOffsets,
			startupOptions.startupTimestampMillis);
	}

而在UpsertKafkaDynamicTableFactory中却是写死的参数：

return new KafkaDynamicSource(
			schema.toPhysicalRowDataType(),
			keyDecodingFormat,
			new DecodingFormatWrapper(valueDecodingFormat),
			keyValueProjections.f0,
			keyValueProjections.f1,
			keyPrefix,
			KafkaOptions.getSourceTopics(tableOptions),
			KafkaOptions.getSourceTopicPattern(tableOptions),
			properties,
			earliest,
			null,
			0,
			true);

结合之前修改flink1.11.1的经验修改：

1.修改UpsertKafkaDynamicTableFactory中optionalOptions的语义识别：

@Override
	public Set<ConfigOption<?>> optionalOptions() {
		final Set<ConfigOption<?>> options = new HashSet<>();
		options.add(KEY_FIELDS_PREFIX);
		options.add(VALUE_FIELDS_INCLUDE);
		options.add(FactoryUtil.SINK_PARALLELISM);
		options.add(SCAN_STARTUP_MODE);
		options.add(SCAN_STARTUP_SPECIFIC_OFFSETS);
		options.add(SCAN_TOPIC_PARTITION_DISCOVERY);
		options.add(SCAN_STARTUP_TIMESTAMP_MILLIS);
		return options;
	}

2.修改UpsertKafkaDynamicTableFactory中KafkaDynamicSource部分的传参：

final StartupOptions startupOptions = getStartupOptions(tableOptions);

		return new KafkaDynamicSource(
			schema.toPhysicalRowDataType(),
			keyDecodingFormat,
			new DecodingFormatWrapper(valueDecodingFormat),
			keyValueProjections.f0,
			keyValueProjections.f1,
			keyPrefix,
			KafkaOptions.getSourceTopics(tableOptions),
			KafkaOptions.getSourceTopicPattern(tableOptions),
			properties,
			startupOptions.startupMode,
			startupOptions.specificOffsets,
			startupOptions.startupTimestampMillis,
			true);

修改完毕，打包验证

验证：

指定sql语义：

with (
  'connector' = 'upsert-kafka',
  'topic' = 'TOPIC_TEST',
  'properties.bootstrap.servers' = '127.0.0.1:9092',
  'scan.startup.mode' = 'earliest-offset',
  'scan.startup.specific-offsets' = 'partition:0,offset:42',
  'key.format' = 'json',
  'value.format' = 'json'

);

查看运行结果，完美解决。

数据脱敏，就不展示了