ExoFlow: A Universal Workflow System for Exactly-Once DAGs
summary
ExoFlow implements a workflow runtime that decouples execution from recovery, providing exactly-once semantics via a separate recovery layer, task annotations for nondeterminism and external outputs, and references for internal outputs so large data (e.g., Spark RDDs) can be shared instead of copied. Prototype runs on an AWS Lambda backend and a centralized controller. Experiments (ETL + serverless trip reservation) show up to ~50% lower P99 latency versus a recent serverless workflow system and correct rollback/commit behavior.
AI-written summary · about
chapters (6)
- 0:00Introduction to workflows
- 2:17Recovery trade-offs
- 3:25The case for ExoFlow
- 6:32ExoFlow design principles
- 10:31Architecture and performance
- 12:55Experimental results
related talks
01h 00mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer?
30mLightflow - A lightweight, distributed workflow system
52mEvent Driven Workflows
20mDataflow Engines for Executing Cloud Applications: a Maslow's Hammer or Natural Outcome?
42mHow to build distributed systems in JavaScript
37mUsing Elixir for scheduling cloudnative DAG workflows
51mComplex Event Flows in Distributed Systems
24mStateful Functions: Flexible like a Microservice, Consistent like a Monolith