Genesis: Terraforming a New Home for Firefox Crash Reporter
summary
Migrates the Socorro Firefox crash-reporter off a single datacenter into AWS, builds an automated continuous delivery pipeline with peer review, infra-as-code and monitoring, and replaces HBase-backed pieces while keeping Elasticsearch and Jenkins as stateful exceptions. Monitoring reveals a surprise double-count bug that looks like 60→30 req/s double-count during the first cutover. Uses Terraform + AWS + CloudFormation in practice, requires a second migration, and ends up running in EC2 with faster deploys but remaining limits: API/account quotas, no parallel testing, and hairy stateful services.
AI-written summary · about
chapters (21)
- 0:00Intro
- 1:20IN ALL SERIOUSNESS...
- 2:52A BRIEF HISTORY OF EVERYTHING
- 4:08THERE WERE A LOT OF MOVING PARTS
- 4:49FAILURE IS AN OPTION
- 5:44CEPH: A VERY SHORT STORY
- 6:07THE END IS NIGH!
- 6:57GENESIS
- 10:03THE (OLD & BUSTED) HUMAN STACK
- 11:03THE (NEW HOTNESS) HUMAN STACK
- 11:59THINGS WE WANTED
- 12:51WHAT IT COULD LOOK LIKE
- 13:50WHO WATCHES THE WATCHERS? (OR, A WATCHED SERVICE NEVER FAILS.)
- 15:14IT'S SCIENCE-Y
- 16:13PRACTICALLY SPEAKING
- 17:04MIGRATION SAGA: PART ONE
- 18:22MIGRATION SAGA: PART TWO
- 18:58HIGH DEGREES OF CONFIDENCE
- 19:25IT'S NOT ALL SUNSHINE AND RAINBOWS
- 21:08IT'S MOSTLY SUNSHINE AND RAINBOWS THO
- 21:37THE GRITTY DETAILS
related talks
29mBreaking Down the TofuLith. How We Reduced Lead Times for Multi-Tenant Infra Changes...
42mPlanning To Fail
27mCase Study: Re-Thinking Our Infrastructure Tooling
21mAvoiding That $1M Dollar AWS Bill - Mark Michael, Glenn Oppegard
29mEverything, Everywhere, All at Once: Terraform and Indeed's Move to AWS
30mLeaping from Mainframes to AWS: Technology Time Travel in the Government
58mAchieving unparalleled elasticity on AWS at the lowest cost
30mTerraforming RDS: What Instacart has learned managing over 50 AWS RDS Postgres instances with Terr