【极数系列】Flink项目入门搭建（03）
【极数系列】Flink项目入门搭建（03）
引言
gitee地址：https://gitee.com/shawsongyue/aurora.git
源码直接下载可运行，模块：aurora_flink
Flink 版本：1.18.0
Jdk 版本：11
1.创建mavenx项目
2.包结构
3.引入pom依赖
tips：transformer处写主启动类
4.0.0
com.xsy
aurora_flink
1.0-SNAPSHOT
11
3.8.1
UTF-8
UTF-8
1.2.75
2.17.1
1.18.0
2.11
2.17.1
com.alibaba
fastjson
${fastjson.version}
org.apache.flink
flink-java
${flink.version}
org.apache.flink
flink-streaming-scala_2.12
${flink.version}
org.apache.flink
flink-clients
${flink.version}
org.apache.logging.log4j
log4j-slf4j-impl
${log4j.version}
org.apache.logging.log4j
log4j-api
${log4j.version}
org.apache.logging.log4j
log4j-core
${log4j.version}
${project.name}
src/main/resources
src/main/java
**/*.xml
org.apache.maven.plugins
maven-shade-plugin
3.1.1
package
shade
org.apache.flink:force-shading
org.google.code.flindbugs:jar305
org.slf4j:*
org.apache.logging.log4j:*
*:*
META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA
org.xsy.sevenhee.flink.TestStreamJob
org.springframework.boot
spring-boot-maven-plugin
${spring.boot.version}
true
${project.build.finalName}
repackage
maven-compiler-plugin
${maven.plugin.version}
${java.version}
${java.version}
UTF-8
-parameters
aliyun-repos
https://maven.aliyun.com/nexus/content/groups/public/
false
aliyun-plugin
https://maven.aliyun.com/nexus/content/groups/public/
false
4.增加log4j2.properties配置
tips：resource目录下增加该配置，主要用于日志打印
rootLogger.level=INFO
rootLogger.appenderRef.console.ref=ConsoleAppender
appender.console.name=ConsoleAppender
appender.console.type=CONSOLE
appender.console.layout.type=PatternLayout
appender.console.layout.pattern=%d{HH:mm:ss,SSS} %-5p %-60c %x - %m%n
log.file=D:\\tmp
5.创建主启动类
tips：编写了一个简单的有界数据流处理demo程序
- step1：创建flink程序运行所需环境
- step2：创建数据集
- step3：把有限数据集转换为数据源
- step4：简单通过flatmap处理数据
- step5：输出最终结果
- step6：启动任务
package com.aurora;
import org.apache.flink.api.common.functions.FlatMapFunction;
import org.apache.flink.streaming.api.datastream.DataStreamSource;
import org.apache.flink.streaming.api.datastream.SingleOutputStreamOperator;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
import org.apache.flink.streaming.api.functions.sink.SinkFunction;
import org.apache.flink.util.Collector;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.ArrayList;
import java.util.Random;
import java.util.UUID;
/**
* @author 浅夏的猫
* @description 主启动类
* @date 22:46 2024/1/13
*/
public class Application {
private static final Logger logger = LoggerFactory.getLogger(Application.class);
public static void main(String[] args) throws Exception {
//1.创建flink程序运行所需环境
StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//2.创建数据集
ArrayList list = new ArrayList<>();
list.add("001");
list.add("002");
list.add("003");
//3.把有限数据集转换为数据源
DataStreamSource dataStreamSource = env.fromCollection(list).setParallelism(1);
//4.简单通过flatmap处理数据，
SingleOutputStreamOperator flatMap = dataStreamSource.flatMap(new FlatMapFunction() {
@Override
public void flatMap(String record, Collector collector) throws Exception {
//数据追加随机数
String uuidRecord=record+ UUID.randomUUID().toString();
//当前环节处理完需要传递数据给下个环节
collector.collect(uuidRecord);
}
});
//5.输出最终结果
flatMap.addSink(new SinkFunction() {
@Override
public void invoke(String value) throws Exception {
logger.info("当前正在处理的数据:{}",value);
}
}).setParallelism(1);
//6.启动任务
env.execute();
}
}
6.构建打jar包
7.flinkUI页面部署
1.点击add new上传对应的应用包
2.主类填写com.aurora.Application
3.检查任务running状态，大概几秒钟跑完
