Implement Reliable, Isolated & Unified Job Submission

on YouTube ↗30m

" summary

A platform running over Flink manages ~1,200 jobs that process ~35 trillion messages and ~4.5 PB/day and replaces flaky ad hoc submission with an atomic protocol: job graph persistence in the job-graph store marks successful submission, only the dispatcher writes the job-registry to avoid races, and finished states are archived rather than eagerly cleared to prevent duplicates; this uses transactional high-availability services, adds an application mode for server-side compilation and dependency localization, and exposes layered clients (deployer, cloud client, job client).

AI-written summary · about

" chapters (24)

1. 0:00Intro
2. 0:30Outline
3. 0:57Oceanus: Tencent Real-Time Compute Platform
4. 2:13Cluster Deployment(Session Mode)
5. 3:20Job Submission(Session Mode)
6. 6:30Cluster Deployment(Job Mode)
7. 8:34Failure Case: Job Manager failed to start in time
8. 9:38Failure Case: Job Manager crashed after job finished
9. 11:25What to define job submitted & executed
10. 12:56How to achieve atomic job submission
11. 16:09Drawbacks
12. 18:24Application Mode: Package User Program
13. 19:20Application Mode: Prepare Dependencies
14. 20:10Application Mode: Cluster Deployment
15. 20:25Application Mode: Download Dependencies
16. 20:53Application Mode: Execute in Isolation
17. 21:38Deployment: Recap
18. 23:07Client Interface: The Problem
19. 24:01Client Interface: Three-Layers Architecture
20. 25:08Client Interface: Deployer
21. 25:56Client Interface: Cluster Client
22. 27:03Client Interface: Job Client
23. 28:04User Case: Platform
24. 29:12User Case: End-User

" related talks

• 38mBuilding Unified Streaming Platform at UberZhenqiu Huang, Venkata Sanath Muppalla · Flink Forward Global Virtual 2020
• 40mGrowing a managed Flink streaming platform from scratchChen Qin · Flink Forward Global Virtual 2020
• 37mBuilding and operating large-scale streaming applications at KingGyula Fora · Flink Forward Berlin 2017
• 35mApache Flink 2.0: Unified Enterprise Data Processing System and BeyondBowen Li · Scale By The Bay 2019
• 43mStream Processing and Applications in the Modern AgeStephan Ewen · Flink Forward Europe 2019
• 44mFlink’s New Batch ArchitectureTill Rohrmann, Ma Guowei · Flink Forward Europe 2019
• 37mKeynote | Flink Ahead: What Comes After Unified Batch & Streaming?Stephan Ewen · Flink Forward Global Virtual 2020
• 24mStateful Functions: Flexible like a Microservice, Consistent like a MonolithSeth Wiesman · Flink Forward Global 2021