Design (Failures) Case Study
summary
Circonus describes building a custom C message queue and a bespoke columnar store (IronDB) to replace RabbitMQ and Postgres, achieving ~1M msgs/sec per instance, cross-continent failover, and histogram-first storage. Storage uses LMDB + LZ4 and packs points to ~8 bits per point; stream analytics first ran in-queue then was moved into a C/Lua broker for ~1000x lower latency (17 ms → 10 µs). Failures include conflating control/data planes, hidden GC bugs, and underestimating ops and observability needs; the rework yields HA, tracing, and operability.
AI-written summary · about
chapters (16)
- 0:00Start
- 2:29Message Queues
- 3:44The failure of RabbitMQ
- 5:08Step 2
- 13:1511 months of safety
- 14:06Histogram instrumentation
- 15:02Zipkin instrumentation
- 18:20Feature: Stream Analytics
- 19:15First Approach: Polling the DB
- 20:49Stream Analytics (2015): Beaker
- 23:16The Regrets of Beaker
- 25:02Regrets of Beaker: Cultural Aspects 2015
- 28:33CAQL Broker: Observability is Key
- 31:26Log Analysis: Poor man's Splunk / ELK
- 32:13CAQL Broker: High Availability
- 36:23Lessons Learned
related talks
21mBuilding Distributed Platforms at Extreme Scale on Fast Changing Data
40mLatency, Throughput & Fault Tolerance: Building a Modern Database
29mApplying (D)DDD and CQ(R)S to Cloud Architectures
35mQuestioning Database Claims: Design Patterns of Storage Engine
36mDesigning for failure - architecting fault-tolerant system
34mDistributed Systems - What Can Go Wrong and How to Get It Right
36mTelemetry Analysis for Scaling
57mDSS Keynote — Back to the Future: Prepare Your Data Infrastructure for Anything