~/talk

Designing for Failure: How to Manage Thousands of Hosts Through Automation

on YouTube 28m
Talkat LISA18 · on Monday, 2018-10-29· at 14:00· Legends Ballroom ABC· [devops-and-delivery]

speaker

name:Brandon Bercovich
links:none on file

resources

video:https://youtu.be/dgYaolkfPZ4
slides:none on file

summary

Uber runs thousands of hosts with Apache Mesos and Aurora and builds a Cluster Lifecycle Manager to automate safe upgrades, drainage and remediation. CLM stores goal-state in Git, queries a local graph cache of inventory and monitoring, and dispatches workflows to Cadence orchestration workers (Cassandra-backed) that rate-limit batches, enforce SLAs (example: maintain 90% replicas for 10 minutes) and escalate from reboot to RMA. Early rollouts (batch sizes like 250) caused outages; CLM’s service-aware checks and automation cut hundreds of manual host alerts to mostly automatic fixes.

AI-written summary · about

chapters (4)

  1. 0:00Start
  2. 0:12Designing for Failure
  3. 5:11Fleet Management
  4. 10:45Host Management

related talks

~/search
type to search 6,249 conferences, 138,987 speakers and 185,172 talks
~/help
cmdctrl k
search every conference, speaker and series
/
focus the filter search
h
home
c
conferences
s
speakers
e
series
t
light / dark — the status bar button does the same
?
this help
esc
close this, then clear every filter — or just the search, while you are typing in it

Ignored while you are typing. Pages without a filter rail have no search to focus.