Orchestration of Big Data systems: A survivor's guide
summary
Describes concrete tradeoffs: arrangement is handled by Spark/Impala so orchestration focuses on coordination and dependency management, not deployment. Compares Oozie XML verbosity, Azkaban plugin model and job runners like Livy and Spark Job Server that lack dependency management and HA but offer reusable, parameterized jobs. Details building two custom orchestrators that use Akka then switch to Kafka for Kafka at-least-once delivery, suffer split‑brain and deployment pains, and concludes custom orchestration works only when platform tools cannot support needed job types, backpressure or defensive HA.
AI-written summary · about
chapters (21)
- 0:00Start
- 1:07Orchestration: what is it?
- 1:57General orchestration: Arrangement
- 2:46General orchestration: Coordination
- 3:23General Orchestration: Tools
- 5:18Big Data Orchestration: arrangement
- 7:26Big Data Orchestration: data pipelines
- 8:30Big Data Orchestration: data dependencies
- 9:16Big Data Orchestration: availability/fault tolerance
- 11:03Ad-hoc orchestration
- 13:10Integration buses/BPMN: NOT FOR BIG DATA
- 14:11Orchestrators: Oozie
- 15:50Orchestrators: Azkaban
- 19:01Spark Job Runners: Common parts
- 20:21Spark Job Runners: Specifics
- 23:50Stop universe building!
- 26:27Custom Orchestration: why?
- 28:10Custom Orchestration: Architecture
- 29:37Custom Orchestration: Technology
- 32:41Custom Orchestration: Results
- 33:15Summary
related talks
26mModern Data Orchestrators
47mAll the DataOps, all the paradigms
02h 18mDistributed Systems: In the Daily Life of the Data Engineer
29mBig Data Analysis
59mStreaming Big Data w/Spark, Spark Streaming, Kafka, Cassandra & Akka
26mRe-architecting the Data Platform with Spark on Kubernetes
21mAvoiding Spark Pitfalls at Scale
04mInterview