Deep Dive: Azure Resource Manager Outage
summary
A legacy configuration file missing in production (present in tests via a transitive dependency) causes a reflection-based static initializer to throw a TypeInitializationException that is fatal in .NET, crashing worker slots. The rollout via feature flags hits paired regions and, after removing experimental v-next workers, reduces resiliency, extending impact to 35 hours and about 3% of requests. The worker loop used task WhenAll for long-running slots so crashes batten down capacity slowly. Mitigations: targeted rollback, separated investigation team, longer metadata TTLs, remove static initializers, tighten test artifacts and telemetry.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 0:22Overview
- 2:42What to Expect
- 3:22System Overview
- 6:35Outage Detection
- 13:48Timeline
- 16:41Rollback
- 18:30Understanding the outage
- 19:00Oncall expectations
- 20:08Why is West US impacted
- 21:29Statistical analysis
- 23:20Why were these errors delayed
- 24:44Stack trace
- 25:03Demo
- 26:51Worker Slot
- 28:11Work Dispatcher
- 30:06Poison Messages
- 31:05Improving Resource Manager
- 35:06Closing Thoughts
related talks
01h 00mDiagnosing issues in ASP.NET Core Applications
46mKickstart your .NET modernization journey with the RWA pattern
49mWar stories from .NET team
16mInterview Speaker
11mTotal Rewrite: A Story About F# and Azure Functions in Production
01h 02mWriting high performance code in .NET
47mHow We Found a 2 GB/Day Leak in 5 Minutes – dotMemory Meets Akka.NET
48mEngineer tested, manager approved: Migrating Windows/.NET services to Linux