Systematising Production Operations: Reddit Control Tower
summary
Built a self-service platform called Reddit Control Tower that turns manual prod steps into auditable admin APIs; components are CLI + admission server + control-tower servers with federated secrets and Octa auth over VPN. Initial v0 implements one white‑glove operation, then scales to two new control towers and a handful of ops; cluster agents run in prod clusters for deep changes. Traffic shifts and rollouts use it heavily in incidents, producing canonical events, Slack notifications, and fewer error‑prone manual steps.
AI-written summary · about
chapters (9)
- 0:00Reddit Control Tower intro
- 1:03Manual ops challenges
- 3:49Admin APIs solution
- 5:12Control Tower architecture
- 8:43Admission server access
- 9:48Development framework
- 10:50Early results adoption
- 11:53Traffic management success
- 13:39Key takeaways future
related talks
32mKubernetes at Reddit: Tales from Production
22mRaise a Production System:Making on-call human-scale w/ Observability & Trust
01h 02mTools for simplified large scale systems
01h 39mPostOps: A Non-Surgical Tale of Software, Fragility, and Reliability
39mAutomating reporting on compliance controls at cloud scale
23mMore Power, Less Pain: Building an Internal Platform with CNCF Tools
28mStay in Control While Moving Fast
48mDevops@Scale