Hacking the Pachyderm: Scaling Servers and People
summary
Community instance grows 700 to 30,000 users in weeks, triggers transatlantic migration of a distributed Mastodon stack with zero data loss. Ops details: single-machine NFS/Postgres/Redis bottlenecks, observed ~300 IOPS, and pathological scaling when connections hit 4098 Postgres connections. Response mixes quick fixes (move logs to RAM, move DB off the basement host), process changes (public incident reviews, documented runbooks), and platform work using NixOS and Terraform Cloud; volunteers maintain service without burnout.
AI-written summary · about
chapters (6)
- 0:00Introduction and context
- 2:04Philosophy of system failure
- 5:23The infrastructure challenges
- 12:51Debugging the performance
- 18:56Culture and team management
- 24:43Operational best practices
related talks
54mGetting to 300 million requests a month with a one man AI/Dev-Ops team
48mHow to Tame A Mastodon: Lessons for PostgreSQL at Scale
47mDeploying Like a Boss: From 2 Servers to 20,000
47mScaling: from 0 to 25 million users
44mIntroduction to application scaling
33mHerding Elephants
23m"In the Belly of the Whale: Operations at Twitter"
55mDevOps @Scale (Greek Tragedy in 3 Acts)