Stress Testing in Production: The New York Times Engineering Survival Guide
summary
Describes a production stress test using JMeter + BlazeMeter against a production‑mirror environment that simulates newsroom writes and user reads; test uncovers 100% 500 errors for a service, exhausted cloud quotas, suboptimal autoscaling rules and high latency, teams fix issues and rerun the test. Execution includes synchronized multi‑executor steps and identifiable test traffic headers. On election night the site absorbs up to 40x sustained traffic, serves results with only minute‑scale outages and measurable business uplift.
AI-written summary · about
chapters (18)
- 0:00Introduction to NYT scale
- 1:09Election traffic spikes explained
- 2:13Challenges for midterm elections
- 3:18Operation Election Readiness
- 4:19Stress testing goals
- 5:43What is stress testing
- 7:49Key stress testing objectives
- 9:33Planning the stress test
- 11:40Preparing for production stress test
- 13:26Designing the stress test
- 16:59Location-based traffic simulation
- 18:24Monitoring during stress tests
- 19:28Executing the stress test
- 21:56Stress test results and findings
- 23:41Key improvements after testing
- 25:29Midterm elections day results
- 26:38Lessons learned and future plans
- 27:43Value of stress testing
related talks
41mMartijn Goossens: Become a Performance Test Master - the Non-Technical Side of Things
35mStress Testing as a Culture
34mStress-Test tools in Azure to optimize your reliability
26mLoad testing websites following the steps of a user journey
47mLoad Testing You Always Wanted
48mHow to not Crash in Prod, Load Testing Crash Course
53mEarly Performance Testing: Finding News You Can Use
33mPerformance Testing in Agile and DevOps