Building and operating large-scale streaming applications at King
summary
King runs Apache Flink jobs that consume over 40 billion events per day and maintain multi‑TB user state (publisher lists over 10 TB). A single Flink backend job multiplexes user events to dynamically deployed scripts stored via Kafka and exposed through the Arbor/Arigato API; state sits in RocksDB with incremental checkpoints to HDFS, deployed on YARN. The stack adds a dedicated aggregate topic, safe‑point/version management, testing pipelines that simulate deployments and failures, and extensive metrics and alerting.
AI-written summary · about
chapters (11)
- 0:00Introduction and company background
- 3:53Types of streaming applications
- 7:51Real-time analytics platform overview
- 13:44Evolution of state backends
- 17:38Handling dashboards and aggregates
- 21:09Testing the platform
- 23:16Scaling the infrastructure
- 25:05Application management in Flink
- 27:16Metrics and monitoring
- 29:03Self-regulating streaming systems
- 32:17Closing and Q&A
related talks
50mPanel: Large Scale Streaming in Production
35mApache Flink 2.0: Unified Enterprise Data Processing System and Beyond
38mBuilding Unified Streaming Platform at Uber
40mGrowing a managed Flink streaming platform from scratch
42mApache Flink deep-dive #bbuzz
37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?
36mBuilding a Self-Serve Stream Processing Platform with Flink
35mLessons learned on Apache Flink application availability in a hosted Flink service