Keeping Developers Happy with a Fast CI

on YouTube ↗28m

" summary

Shopify runs ~210,000 Ruby tests (adding ~30,000/year), executes ~1,000 builds/day and ~100M test runs/day, and treats CI as production: Buildkite + Google Kubernetes agents scale on demand, timings stream to Kafka and a data warehouse, slow tests get aggressive timeouts or disabled (disabling ~10 tests cuts ~10 minutes), and dynamic mapping uses rotoscope + mini test reporter to build file→test maps so PRs run ~40% of tests on average (≈85,000), which noticeably speeds feedback.

AI-written summary · about

" chapters (8)

1. 0:00Intro
2. 3:11Data Driven Development
3. 6:22Test Optimization
4. 8:47World Test Suite
5. 9:22Test Selection
6. 13:41Fixtures
7. 15:06Test Prioritization
8. 22:21Conclusion

" related talks

• 21mTesting Rails at ScaleEmil Stolarsky · RailsConf 2016
• 34mShitlist-driven development and other tricks for working on large codebasesFlorian Weingarten · RedDotRubyConf 2017
• 43mA day in the life of 2,000 developersJames Carr · Friendly.rb 2024
• 32mBuilding Continuous Integration and FeedbackChris McFee · DevOps Enterprise Summit London 2019
• 39mScaling R&D to thousands of people without dedicated QAAdam Archer · Assert(js) 2019
• 35mApplying SRE Principles to CI/CDMel Kaulfuss · Euruko 2022
• 30mKeith and Mario's Guide to Continuous Deployment with RailsKeith Pitt · RailsConf 2014
• 47mHow to win in software development with tighter feedback loopsBrian Lucas · Breakpoint 2020