OpenShift CI: What if we stopped retesting everything all the time?
summary
OpenShift CI uses prow jobs and Tide retest to speculatively re-run PRs when base changes, but 2/3 of PRs merge without any retest and only 0.8% see additional revisions during retesting. Busy repos like origin merge immediately only 10% of PRs; batch success rates vary (origin ≈75%, HyperShift ≈50%, console ≈27%, MCO ≈20%). Longest-job latency governs throughput — a 4-hour longest job yields ~one merge per 4 hours — so the concrete proposal is: skip repeated expensive flaky end‑to‑end runs and only retest fast stable jobs, relying on TRT post‑merge reverts as backstop.
AI-written summary · about
chapters (12)
- 0:00OpenShift CI challenges
- 1:46Dreaded tide retest
- 4:34Why retesting happens
- 8:06Costs of naive retesting
- 12:20How Tide handles retests
- 15:31Batch merges in Tide
- 18:21Failures kill merge throughput
- 22:37Gains from retesting
- 26:33Binary nature of retest decisions
- 28:40Retest effectiveness data
- 31:06Trade-offs and improvements
- 35:16Q&A
related talks
49mCI for OpenShift: Prow, ci-operator and the future
22mCI that CIs itself: Rehearsals in OpenShift CI
36mThinking like a site reliability engineer to improve CI
42mCI/CD - Let The Robots Do Their Job!
22mKeeping builds green at scale
26mA State of Continuous Merge: The Secret to More Productive Dev Teams
36mShifting Performance Engineering Left
13mOptimizing Workload Management in a Multi-Cluster Environment: A Case Study