Production-Ready Flink and Hive Integration - what story you can tell now?
summary
Flink 1.10 ships a production-ready Hive integration: a Hive catalog that persists Flink table metadata across HMS versions, reads and writes Hive tables views partitions with vectorized ORC reader, partition pruning, projection and limit pushdown, and writes static and dynamic partitions. It reuses Hive UDFs and ~400 Hive built-ins via a Hive module. Flink 1.11 adds a streaming file sink for near‑real‑time Hive ingestion, native Parquet reader, packaged Hive dependencies and a Flink JDBC gateway; beware small‑file commits (practical commit interval ~5–10 minutes).
AI-written summary · about
chapters (23)
- 0:00Intro
- 1:15Agenda
- 1:50Background
- 2:40Goals
- 6:05Hive Metastore
- 6:55Hive Catalog
- 8:02Hive Catalog Example
- 8:35Flink SQL CLI Setup
- 9:45Kafka Table Setup
- 10:12Hive CRI
- 11:30Flink SQL CLI
- 12:40High version support
- 13:10Reuse Hive UDFs
- 15:40Reading Hive tables
- 16:32Writing Hive tables
- 16:57Flink 1111
- 18:23Dependency Management
- 19:54Use Cases
- 25:07Poll
- 26:44Unsolved Problem 1
- 28:42Unsolved Problem 2
- 31:49Bonus
- 33:34Conclusion
related talks
36mFlink SQL: Unified Streaming and batch for hive integration
40mUnify Enterprise Data Processing System: Platform-level integration of Flink and Hive
37mSQL Ask-Me-Anything
42mApache Flink deep-dive #bbuzz
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
41mBeyond the Hype: When Apache Flink Solves Real Problems
42mWhat's new for Flink's Table & SQL APIs? Planners, Python, DDL
37mThe Future of Apache Flink