Catching Bugs Before They Bite: How Zillow Leverages Shadow Testing for Safer GraphQL Rollouts
summary
Zillow builds a production-grade shadow testing platform called ShadowRunner that mirrors real production GraphQL traffic (45–55B requests per 30 days, 15k–20k req/s) to run control vs treatment experiments, diff responses and latencies, and surface regressions. It supports URL and header flows, subgraph and supergraph experiments, alias rewriting to avoid PII leaks, and an explicit allow list for fields. Daily ETL jobs produce UI reports that caught a trìmódal latency shift on Apollo Router and subtle boolean mismatches, replacing weeks of manual spreadsheet diffs and preventing production incidents.
AI-written summary · about
chapters (11)
- 0:00Safe GraphQL deployments
- 3:32Shadow testing importance
- 7:26Rehearsal versus production
- 9:12Early Shadowrunner success
- 11:22Manual to platformized Shadowrunner
- 14:12Shadowrunner architecture
- 19:28Data handling safeguards
- 22:43Shadowrunner UI walkthrough
- 24:50GraphQL migrations impact
- 27:58Shadowrunner future plans
- 28:59Closing and call to action
related talks
19mFunctional Testing of GraphQL APIs: Simple Checks That Catch Real Bugs
35mGraphQL in real life
08mModern GraphQL API Security Testing
40mGraphQL: A Case Study
38mPerformance Testing sucks! How to get started testing RESTful APIs and GraphQL
25mSophisticated Schema Mocking
28mHow Not-To Test in Production: Learnings from DoorDash
15mNavigating GraphQL Vulnerabilities: SRE / DevSecOps