Monoceros: Faster and Predictable Services through In-pod...
summary
Built a Go multiprocess controller called Monoceros that replaces Ruby Einhorn, initially using SO_REUSEPORT then running a proxied data path with Envoy. Deployments run at Reddit scale and add per-worker CPU metrics plus timed worker restarts to combat leaks, producing roughly 1 TB memory saved across the fleet. Measured results show 20–30% P99 latency reduction at ingress and ~10% P90 improvement for critical home-feed queries, plus much lower CPU variance and more predictable scaling.
AI-written summary · about
chapters (14)
- 0:00Reddit's infrastructure challenges
- 1:27Python's global interpreter lock
- 2:55Problems with Einhorn and initial solutions
- 3:38Monoceros design goals
- 4:21Building the minimum viable product for observability
- 5:24Memory leaks and restarts
- 6:48Observability and CPU metrics
- 7:31Proxy mode for load balancing
- 8:56High-traffic performance results
- 10:00Critical request latency gains
- 10:40Predictable CPU scaling
- 11:21Socket and FD handling
- 12:25Pod-level load balancing
- 12:45Open-sourcing Monoceros
related talks
27mScaling Ruby Applications with Mesos and Docker
39mHow we learned to love the Data Center Operating System
19mKeynote: Operating 600+ Mesos Servers on 7 Datacenters @Criteo
28mMoving a Monolith from AWS EC2 to K8
24mMicrofrontends
38mHighly Scalable Consumer Applications using Cloud Native Open Source Tools
17mBuilding a gRPC Proxyless World: How Reddit Scaled Resilience...
35mMigrating the Next Generation Mobile Core Towards 5G with Kubernetes