How Not-To Test in Production: Learnings from DoorDash
summary
Company abandons flaky staging and runs test and prod traffic on one stack claiming lower ops cost and higher fidelity; at DoorDash scale (thousands of merges per day, ~700 backend engineers) teams push synthetic load for 2x peak capacity checks, route test calls via OpenTelemetry headers, and isolate state with an in‑house database mesh that proxies SQL over gRPC; service mesh (Envoy), Kafka/Redis shims, and leak‑detection tooling are built and iteratively rolled out — approach works but context propagation and data isolation require ongoing engineering.
AI-written summary · about
chapters (14)
- 0:00Testing in production intro
- 1:32DoorDash tech stack
- 2:58Problems with dedicated testing environments
- 4:45Why choose prod testing
- 7:40How DoorDash implemented testing in production
- 10:53Key challenges
- 14:26Testing with sandboxes
- 16:14Data isolation for capacity
- 18:43Adoption and retrofitting
- 20:33Context propagation issues
- 21:56Common pitfalls
- 24:28Takeaways and advantages of testing in production
- 25:57What's next for DoorDash's testing approach
- 27:49Q&A
