1.5 Billion Tests Per Day
summary
Claim: a centralized system records about 1.5 billion test results per day and feeds models and tooling that cut cost and noise. A language-agnostic test runner called Test Pilot standardizes execution and reporting to databases and Hive. A predictive test selection model halves test cost while catching ~95% of failures and blocking >99% of bad PRs. Continuous full runs every 20 minutes plus automatic BCT bisection and automatic revert within 14 minutes recover from slipped regressions.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 1:29Testing scales differently at Meta
- 4:03Testing strategies and inverted pyramid
- 7:15Billions of tests daily
- 8:39Test runners and results at scale
- 12:53Relevant tests to run
- 16:28Predictive test selection with ML
- 17:33Flaky tests and impact
- 20:01Cost of flaky tests
- 21:44Android flakiness case study
- 24:37Non-blocking test runs
- 26:04Root-causing and reverting failures
- 28:12Test states and flakiness scores
- 31:20Probabilistic flakiness score
- 32:50Key takeaways
