On a shoe-string and a t2.small: scaling on a [zero] budget.
summary
A small Django/Flask side project called secateur runs on a reserved t3.small and struggles at 250k–2M block actions per day because Amazon EBS throttles when the EBS IOPS burst balance hits zero. Fixes include tuning Postgres (per-transaction synchronous_commit off, wal writer delay), moving Celery broker from Redis to SQS, batching work (one task blocks 50 users), token-bucket per-user rate limits, and pragmatic worker restarts to contain a Celery memory leak.
AI-written summary · about
chapters (20)
- 0:00Introduction
- 2:10Secator
- 2:52User growth
- 4:52Welcome
- 6:17Technical details
- 7:20How it works
- 8:38What causes it
- 10:10A couple things
- 10:25Too many disk rights
- 10:45PostgreSQL
- 11:25Disk Rights
- 12:07Synchronous Commit
- 14:16Postgres Tips
- 16:07Celery
- 16:50Memory
- 19:56Memory leaks
- 20:55User constraints
- 22:22Token bucket algorithm
- 23:44Inspiration from constraints
- 24:38Conclusion
related talks
42mA Nice Problem to Have: Django Under Heavy Load
32m(Probably) a retrospective for Secateur
32mDjango on AWS for chump change
39mHigh Performance Django: From Runserver to Reddit Hugs
47mReal World Django Q&A
19mSwiss Army Django: Small Footprint ETL
38mRails in the Cloud
47mHigh Performance Django at Ten: Old Tricks & New Picks