When Can You Skip a Test? Tracking Test Impact
summary
DataDog’s test impact tool claims to cut CI time dramatically — examples show a 15‑minute suite reduced to 9 seconds for some commits and a goal of ~50% average CI time reduction. Implementation journey: start with coverage API (SimpleCov) which gave 4× slowdown, try trace point (line events) with up to 6× slowdown, then build a C extension using rb_event_hook and rb_profile_frames plus RB_SOURCEFILE to hit ~60% worst‑case overhead; add an allocation profiler (const_source_location) and bytecode ISeq scanning to cover constants, bringing median overhead under 20% and fixing missed dependencies.
AI-written summary · about
chapters (11)
- 0:00Introduction
- 1:29Test impact analysis
- 5:04Approaches to coverage
- 8:35TracePoint limitations
- 10:43Native extension attempt
- 14:40Correctness challenges
- 17:09Allocation profiling
- 21:00Constant access tracking
- 23:08Bytecode analysis
- 25:55Final solution
- 26:38Resources and closing
related talks
35mSmarten Up Your Pipeline and Reduce Test Execution Time｜
58mSpeed up your test suite by throwing computers at it
24mSupercharging CI: Test Suite Optimization for Speed, Stability, and Savings
50mTest Intelligence for Architects
52mTest Impact Analysis. Smarten up your pipeline and reduce t
02h 05mFrom slow to go: Rails test profiling hands-on
44mTest Intelligence: Finding More Bugs in Less Time with AI
51mAre Your Test Slowing You Down?