Flink 中 DataStream / DataSet 与 Table 的相互转换

最近在使用Flink Table API，将 DataStream/DataSet 与 Table 的相互转换方法进行总结。

一、pom.xml导入需要的包

    org.apache.flink
    flink-streaming-scala_${scala.binary.version}
    ${flink.version}
    
        
            log4j
            *
        
        
            org.slf4j
            slf4j-log4j12
        
    


    org.apache.flink
    flink-table-planner_${scala.binary.version}
    ${flink.version}


    org.apache.flink
    flink-table-api-scala-bridge_${scala.binary.version}
    ${flink.version}


    org.apache.flink
    flink-table-common
    ${flink.version}

二、方法导入类。

Table API使用Scala的隐式转换，为了使用Scala的隐式转换，请确保导入

org.apache.flink.api.scala._

org.apache.flink.table.api.scala._

org.apache.flink.streaming.api.scala._

三、DataStream or DataSet to Table

定义数据类型

// data type
case class Order(user: Long, product: String, amount: Int)

3.1、 Register a DataStream or DataSet as Table

val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)

// DataStream
val orderA: DataStream[Order] = env.fromCollection(Seq(
  Order(2L, "pen", 3),
  Order(1L, "rubber", 3),
  Order(4L, "beer", 1)
))

// register DataStream as Table
tEnv.registerDataStream("OrderA", orderA, 'user, 'product, 'amount)

val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = BatchTableEnvironment.create(env)

// DataSet
val orderB: DataSet[Order] = env.fromElements(
  Order(2L, "pen", 3),
  Order(1L, "rubber", 3),
  Order(4L, "beer", 1)
)

// register the DataSet as table
tEnv.registerDataSet("OrderB", orderB, 'user, 'product, 'amount)

3.2、Convert a DataStream or DataSet into a Table

val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)

// DataStream
val orderA: DataStream[Order] = env.fromCollection(Seq(
  Order(2L, "pen", 3),
  Order(1L, "rubber", 3),
  Order(4L, "beer", 1)
))

// convert DataStream to Table
val tableA: Table = tEnv.fromDataStream(orderA, 'user, 'product, 'amount)
// 或者
orderA.toTable(tEnv)

val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = BatchTableEnvironment.create(env)

// DataSet
val orderB: DataSet[Order] = env.fromElements(
  Order(2L, "pen", 3),
  Order(1L, "rubber", 3),
  Order(4L, "beer", 1)
)

// convert DataSets to Table
val tableB: Table = orderB.toTable(tEnv, 'user, 'product, 'amount)

四、Convert a Table into a DataStream or DataSet

4.1、Convert a Table into a DataStream

val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = BatchTableEnvironment.create(env)

// DataSet
val orderB: DataSet[Order] = env.fromElements(
  Order(2L, "pen", 3),
  Order(1L, "rubber", 3),
  Order(4L, "beer", 1)
)

// convert DataSets to Table
val tableB: Table = orderB.toTable(tEnv, 'user, 'product, 'amount)

// convert a Table into a DataStream
val ds1: DataStream[Order] = tableA.toAppendStream[Order]
// 或者
val ds2: DataStream[Order] = tEnv.toAppendStream[Order](tableA)

4.2、Convert a Table into a DataSet

val env = ExecutionEnvironment.getExecutionEnvironment
val tEnv = BatchTableEnvironment.create(env)

// DataSet
val orderB: DataSet[Order] = env.fromElements(
  Order(2L, "pen", 3),
  Order(1L, "rubber", 3),
  Order(4L, "beer", 1)
)

// convert DataSets to Table
val tableB: Table = orderB.toTable(tEnv, 'user, 'product, 'amount)

// convert a Table into a DataSet
val ds1: DataSet[Order] = tableB.toDataSet[Order]
// 或者
val ds2: DataSet[Order] = tEnv.toDataSet[Order](tableB)

 

如果有写的不对的地方，欢迎大家指正。有什么疑问，欢迎加QQ群：176098255