Building a Production Control Plane with Tokio
summary
Production control plane for Aurora Distributed SQL is implemented in Rust on Tokio and actually backfills CMK key metadata across every live cluster, rolling the change out in two weeks with no production incidents. It enforces bounded concurrency with JoinSet and per-intent caps (example bound: ten in-flight operations), races tasks against a 30s timeout using select! and recovers partial state, and validates correctness via Turmoil simulation with deterministic replay and simulated time. Telemetry gaps (Tokio console access, stable task dumps) remain.
AI-written summary · about
chapters (9)
- 0:00Control planes in Aurora
- 0:57What is a control plane
- 2:21Enabling CMK keys on Aurora
- 4:05Simulation testing with turmoil
- 6:33Bounded concurrency with join set
- 10:19Actor pattern for orchestration
- 11:43Fairness policy to prevent starvation
- 13:06Rollout success and key takeaways
- 14:30Telemetry and monitoring challenges
