How Netflix Provisions Optimal Cloud Deployments of Cassandra
summary
Shows a concrete capacity planner that converts vague developer inputs (intervals like 1k–100k QPS, 100 GB–1 TB) into cloud Cassandra clusters by: profiling instance types and disk latency distributions with synthetic benchmarks (biosnoop → fitted beta), converting inputs to generative distributions, then running a least-regret Monte Carlo optimizer across instance shapes. Examples: picks 12 m5d xlarge (~$9k/yr) for a 10k-QPS/100 GB workload or 48 i3en xlarge (~$73k/yr) for heavy writes. Post-provision checks use CPU queuing, read-IO hit rates and GC/flush frequencies to validate and scale.
AI-written summary · about
chapters (33)
- 0:00Introduction
- 1:13Agenda
- 2:01Workload Capacity Model
- 2:30Understanding Hardware
- 4:15Hardware Profiles
- 4:26Lab Label Database
- 4:50Hardware Analysis
- 5:25Lifecycle
- 6:08Alpha
- 9:31Pricing
- 9:46Pricing Lifecycle
- 10:21Kubernetes Model
- 10:56Key Insight
- 12:38Service Tier
- 13:22Query Pattern
- 14:10Data Shape
- 16:52Inputs Check
- 17:26Single Function
- 21:17Inputs
- 23:01Memory
- 26:12Working Set
- 31:37Monte Carlo Simulation
- 33:00Regret
- 34:07Example
- 35:38Choice of Constants
- 36:08Least Regretful Choice
- 36:55Sampled Inputs
- 41:08Least Regret Optimizer
- 42:57Metrics
- 44:44Disk Network
- 46:12Buying more memory
- 47:03Conclusion
- 48:00Questions
related talks
46mCassandra Serving Netflix @ Scale
34mHow Netflix Delivers Key-Value and Time-Series Storage at Any Scale
44mTechniques Netflix Uses to Weather Significant Demand Shifts
31mKeeping Your Cloud Footprint in Check
23mArchitecting for Performance, Scalability and Real-time Data with AWS
25mImproving Bad Partition Handling in Apache Cassandra
26mBuilding Financial Systems on Eventually Consistent DBs
28m70 TB Servers – How and Why We Use Cassandra