From Firefighting to Proactive Work: ...
summary
A major MySQL datastore hits a tipping point with a 66% rise in query volume, CPU often near 80% and frequent pages; a rapid firefight mobilizes ~25 engineers, uses Sequel fingerprinting to find bad queries, shifts traffic to replicas and changes schemas, cutting query volume 25% in three days and CPU under 50%. Over months the team defines done metrics (CPU <50%, <10k selects/sec), runs MySQL training and continuous projects, and achieves 80% fewer pages going from ~2 pages/day to ~2/week.
AI-written summary · about
chapters (13)
- 0:00Intro
- 2:27Things started getting worse
- 3:10SQL Fingerprinting
- 4:09Tipping point
- 5:50Work mode
- 7:11A massive effort
- 8:52Huge impact in 3 days
- 10:18New organization
- 12:36Misalignment
- 16:29Great milestone
- 17:43High level of ambiguity
- 18:41Team Week
- 20:00Lessons learned
related talks
25mData Demons: Unleashing Chaos or Driving Performance?
53mMySQL DevOps
32mTackling Slow Queries: A Practical Approach to Prevention and Correction
41mUsing Databases to Pull Your Application Weight
30mFrom 15,000 Database Connections to under 100—A Tech Debt Tale
06mScaling your applications with Cloud SQL for MySQL
01h 02mHow-to MySQL Performance Diagnostic Using Percona PMM
29mAdvanced Optimization and Troubleshooting MySQL Using PMM2