Identifying Hidden Dependencies
summary
Honeycomb deliberately terminates nodes on purpose to test Zookeeper, Kafka, and indexing durability, running controlled restarts every Monday at 3 PM and automatically recycling nodes weekly so no node lives longer than 12 months. Manual experiments uncover regressions like a breaking AWS Ruby library and a Zookeeper client that only contacts zookeeper-0, leading to fixes and to design changes: replay from S3 hourly snapshots instead of rsync between peers, cross-AZ partition placement, and migration of stateful workloads to spot and Graviton2 instances, saving tens of thousands per month.
AI-written summary · about
chapters (43)
- 0:00Intro
- 0:35Big data is operationally complex.
- 0:57Observability is evolving quickly.
- 1:12Two dozen engineers build Honeycomb.
- 1:28We make systems humane to run
- 1:42and empowering engineers.
- 2:34Continuous delivery is an investment.
- 2:49Stable platforms empower innovation.
- 3:06but stateful services can be scary.
- 3:32We need velocity and reliability.
- 3:48Identify potential areas of risk.
- 4:05Prioritize addressing risks.
- 4:24How broken is "too broken"?
- 4:57Service Level Objectives define success.
- 5:31Think in terms of events in context.
- 6:11HTTP Code 200? Latency 100ms?
- 6:27Set a target Service Level Objective.
- 6:38Use a window and target percentage.
- 7:28We keep SLOs at Honeycomb.
- 8:11Alerts usually evaluate every minute.
- 8:39Often, queries come back under 10s.
- 8:56Error budget: allowed unavailability
- 9:11Is it safe to do this risky experiment?
- 9:45Data persistence is tricky.
- 10:06Experiment using error budgets.
- 13:46Infrequent changes.
- 14:25Long-running processes.
- 14:44Data integrity and consistency.
- 15:16Delicate failover dances
- 17:18Restart one server & service at a time.
- 17:50Bugs are shallow with more eyes.
- 18:16Monitor for changes using SLIs.
- 18:39Debug with observability.
- 19:23Test the telemetry too!
- 20:09Verify fixes by repeating.
- 26:42Continuously verify to stop regression.
- 27:17Save money with flexibility.
- 29:44Hypothesize, test, and learn.
- 30:09Celebrate successes and failures.
- 30:38Be more reliable & scalable.
- 31:08Sleep easily at night.
- 31:47You can do this too, step by step.
- 32:34Read our blog! hny.co/blog
