Use Load Testing to Build a Proper Mental Model of Your Service
summary
Intercom builds a custom load generator, hooks the test cluster to production monitoring via terraform, and uncovers concrete failure modes: 35MB queries (1.5M terms) that were previously filtered, coordinated-omission masking unless the generator keeps pounding, and a non-linear scaling cliff around 3.4k qps tipping point where GC pauses and shard imbalance explode latency from 200ms to 10s. Fixes include shard rebalancing alerts, switching disks to RAID0 to avoid fluttering relocations, and documenting SLO-driven red lines for safe capacity planning.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 5:18Load Testing
- 8:07Dashboard
- 10:49Assumptions
- 18:11Outliers
- 20:07Graphs
- 24:11Production Monitoring
- 27:08Low Tests
- 27:52Low Latency
- 28:17Low Disk Usage
- 29:16Documentation
related talks
47mLoad Testing You Always Wanted
27mTesting Before You Scale & Making Friends While You Do It
31mLoad Testing Kubernetes: How to Optimize Your Cluster Resource Allocation in Production [I]
37mExploratory Load-Testing with AI Tooling - it's Time to Ramp Up!
48mHow to not Crash in Prod, Load Testing Crash Course
21mCrunching Millions of Requests for Breakfast: A Tale of Performance Testing Prowess
33mHold My Beer - Load Testing. In Production. On Autopilot.
31mKill! Kill! Die! Die! Load Testing With A Vengeance