Managing to Your SLO Amidst Chaos
summary
Company runs 14 deployments per day while ingesting ~2 million data points/sec and targets four nines for core ingest, using Kafka + S3 and Kafka+S3 replay to guarantee durability. Chaos experiments include killing one Kafka broker weekly and replacing indexing workers to verify recovery; a recent test caught a 12–16 hour restore instead of six, prompting cluster upsizing. Graviton Lambda migration initially increases latency so rollout uses 1% feature-flagged canaries and fast rollback.
AI-written summary · about
chapters (10)
- 0:00Intro
- 2:18Our confidence recipe
- 2:56Measuring reliability
- 6:28How to stay within SLO
- 9:23Validating our expectations
- 13:44Experimenting in prod
- 19:19Not every experiment succeeds
- 26:46Fast & reliable: Pick both!
- 29:05Outro
- 29:36Q&A
related talks
41mScaling Telemetry Systems with Streaming
31mServerless Observability: a case study of SLOs
56mScaling Telemetry Systems with Streaming & Serverless
30mRunning Highly Scalable Tracing, Logging, Monitoring & Metric System in Production
38mTransforming Production Readiness
41mScaling Observability: From Chaos to Clarity with Open Source Tools
58mCultivating Production Excellence
27mObserving Failure in Reactive Systems