How to Make Releases Safer in Baidu
summary
Baidu finds 54% of faults come from changes and moves to staged, package-based rollouts (sandbox → gray → blue) that deploy 1% then 99% of instances to limit blast radius. Rule-based checks cause many false alarms, so an ML-backed anomaly system compares the current-release metric deltas to both a control set and previous releases, using a t-test on deltas across instance sets. On 88 historical releases it flags 15 alerts (9 true, 1 false, 5 due to missing control data), and it stops real incidents and speeds rollback via a one‑button release platform.
AI-written summary · about
chapters (15)
- 0:00Introduction
- 0:49The quantity of release
- 2:29The problem
- 3:19Major problems
- 5:09Minimize losses
- 6:10Build release platform
- 7:13Automatic checking
- 7:51Types of checking
- 8:12Business logic indicators
- 8:56Monitor indicators
- 10:23Intelligent checking
- 14:54How to measure the change
- 16:50Ttest
- 18:03Real Case
- 18:54System Related Indicators
related talks
53mDeliver Results, Not Just Releases: Control & Observability in CD
05mRelease management as a catalyst for CI/CD
21mFrom chaos to control: Future-proofing your Android release process
45mDeploying to production with confidence
07mKeep Calm and Deploy On: Creating Safer Releases with Feature Flags
24mFuture-proofing your Android release process
33mTESTING IN PRODUCTION
26mGreen Checkmarks, Red Flags: What CI/CD Can't Catch