Hacking The Pachyderm: Scaling Servers And People
summary
Hachyderm scales from ~700 to 30,000 users in weeks and migrates a live distributed Mastodon deployment across the Atlantic with lift-and-shift migration to EU, moving media to S3 and adding try_files caching, rebalancing Sidekiq queues and adding human circuit breakers. Root cause is two failed SSDs plus NFS hosting Postgres and media causing cascading I/O failure at ~300 IOPS. Migration succeeds with zero data loss and no burnout.
AI-written summary · about
chapters (16)
- 0:00Introduction to Mastodon
- 2:10What is the Fediverse
- 6:27Scaling crisis timeline
- 9:18Systemic learning over fixes
- 11:26Architecture breakdown
- 16:45Redis and Sidekiq pain points
- 18:55Queue management struggles
- 23:16File system and NFS disaster
- 25:47Emergency migration
- 27:53Why we missed the issues
- 30:01Exponential load factors
- 34:12Systemic adaptation
- 35:40Community ownership
- 37:49Psychological safety
- 40:38Making change healthy
- 41:42Conclusion and takeaways
related talks
24mMigrations under Production Load: How to Switch...
34mTen-year Journey to 10,000 Production Machines
25mWe're Still Down: A Metastable Failure Tale
38mHow we Migrated (...) Without Downtime using Elixir
34mDistributed Systems - What Can Go Wrong and How to Get It Right
18mArchitecting Growth: Scaling Tactics for Prometheus Metrics...
37mSharding: Growing Systems from Node-scale to Planet-scale
18mEngineering Resilience at Scale; Lessons from a Distributed Systems Transformation