Building Koi Pond: Simulating Millions of Slack Clients
summary
Builds a lightweight Go client fleet called koi pond that replaces headless‑Chrome puppets and simulates real websocket connections to exercise Slack's RTM fan‑out; it runs behaviors and formations (frequency‑driven API calls plus coordinated thundering‑herd scenarios), finds real bugs, forces changes to gateway and channel servers and new rate limits, and uses Grafana/Prometheus for KPIs. Cost falls from $0.37 to under a tenth of a cent per client per day and tests hit 1.7M simulated users, later scaling to 2M.
AI-written summary · about
chapters (8)
- 0:00Start
- 5:36High-level overview of Slack's architecture
- 8:15Surface area tested via API Blaster
- 9:10Posting a message to Slack
- 10:42Message propagation mechanism
- 12:31Puppet Show within Slack's architecture
- 23:33Formations (cont.)
- 28:40Were we successful?
related talks
31mKill! Kill! Die! Die! Load Testing With A Vengeance
52mTalk with the Experts
31mRunning Low-Latency Workloads on Kubernetes
33mReaching Ludicrous Speed
36mSimulating Network Conditions with JavaScript
26mIntroduction to Load Testing
27mIn Memory of Travails
25mGame time: A playbook to (unsuccessfully) 10x in a week and (successfully) 10x in a year