Unify Enterprise Data Processing System: Platform-level integration of Flink and Hive
summary
Flink 1.9 ships a platform-level Hive integration that makes Hive metadata and data first-class: a new catalog API and HiveCatalog let Flink read/write partitioned Hive tables, persist non‑Hive metadata into the Hive metastore, and execute most Hive UDFs; 1.10 adds broader Hive version support and projection pushdown. Roadmap items include FLIP‑57 reworking function resolution and a pluggable module system so built‑in Hive functions aren’t reimplemented. The integration is released as part of Flink and is usable today for unified batch+stream processing.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:32Agenda overview
- 0:54Background and motivation
- 4:32Why integrate with Hive
- 9:18Platform-level integration explained
- 10:01Metadata integration in Flink 1.9
- 13:48Catalog API and Hive catalog
- 17:22Hive UDF and source/sink support
- 22:24Flink 1.9 integration summary
- 22:45Flink 1.10 improvements
- 25:03Function catalog rework
- 30:04Pluggable table system
- 32:20DDL and SQL enhancements
- 34:06Future integration plans
- 35:11Conclusion and Q&A
related talks
35mProduction-Ready Flink and Hive Integration - what story you can tell now?
36mFlink SQL: Unified Streaming and batch for hive integration
32mKeynote: Apache Flink - Completing Cloudera’s End to End Streaming Platform
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
43mLeverage the power and simplicity of SQL on Apache Flink
42mApache Flink deep-dive #bbuzz
41mBeyond the Hype: When Apache Flink Solves Real Problems
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?