Building Unified Streaming Platform at Uber
summary
Uber runs 2,500 Flink pipelines handling more than 6 trillion messages per day and builds a unified deployment layer that manages both Streaming SQL and native jobs across per-job clusters. The team ships terablob + kraken for policy-driven blob placement and P2P jar distribution, and leverages Flink 1.11 (FLIP-85) to push job-graph generation into the cluster, enabling >1,000 job launches within one minute. S3 shows an ~80s restart penalty versus HDFS, so a hybrid HDFS/S3 strategy and replication are used to meet zonal-failure and 4‑9s availability goals.
AI-written summary · about
chapters (15)
- 0:00Introduction and agenda
- 1:10Uber’s mission and streaming use cases
- 3:37Existing streaming platforms SNS and FAST
- 6:07Motivation for platform unification
- 8:56Challenges in unifying platforms
- 10:18Infrastructure layer and hybrid cloud
- 12:02Platform layer architecture
- 15:11Ensuring high availability for critical jobs
- 18:23Optimizing job deployment efficiency
- 21:47Reliability across the ecosystem
- 23:31Hybrid cloud storage challenges
- 26:22Storage requirements and S3 evaluation
- 30:13Storage abstraction with Terablob
- 33:25End-to-end deployment workflow
- 36:15Future plans and conclusion
related talks
37mBuilding and operating large-scale streaming applications at King
30mImplement Reliable, Isolated & Unified Job Submission
50mPanel: Large Scale Streaming in Production
36mRunning Flink in production: The good, the bad and the in-between
58mBuilding Self Service Platforms for Apache Flink
35mLessons learned on Apache Flink application availability in a hosted Flink service
46mKeep it going — How to reliably and efficiently operate Apache Flink
36mWhen ordering matters