Mastering Chaos: Achieving Fault Tolerance with Observability-Driven...
summary
Aperture is an open‑source control plane plus data‑plane agent that enforces observability‑driven load shedding using Aperture agent sidecar decisions, Prometheus signals and OpenTelemetry baggage. It implements token buckets, weighted fair queuing and concurrency limits, supports declarative policies expressed as circuit graphs (Rego), and offers precise global quotas via a peer‑to‑peer counting layer. Demo with Hasura + Postgres shows autoscaling from 1→10 replicas and sustained higher acceptance for high‑priority GRPC/GraphQL traffic while low‑priority queries are dropped.
AI-written summary · about
chapters (17)
- 0:00Introduction
- 2:35Little's law and overloads
- 4:19Cascading failure
- 5:18Death spiral
- 6:45Retry storm: permanent overload
- 7:40Metastable failures
- 8:47Common triggers
- 12:50Mitigation with adaptive load shedding
- 13:43Requirements for adaptive load shedding
- 16:36Requirements for prioritization
- 18:18Aperture overview
- 19:38Aperture architecture
- 21:42Adaptive load scheduling policy
- 22:35Load scheduler policy component
- 23:45Adaptive load scheduler insertion
- 25:24Workload prioritization with Aperture
- 28:52Protecting PostgreSQL
related talks
27mAchieving Fault Tolerance in Istio with Observability-Driven Load Management
42mDistributed Monitoring: How to understand the Chaos
36mConfidence with Chaos for Your Kubernetes Observability
25mOpen source observability for private cloud: mission impossible or not?
24mTaming Observability at Scale in a Multi-Cluster Kubernetes Platform at Bloom...
56mFull-stack observability and application monitoring with
12mMastering Observability in Distributed Systems
31mWriting Reliable, Scalable, Fault Oblivious Code On K8s the Easy Way