Scaling HDFS with Consistent Reads from Standby Replicas
summary
Implements consistent reads from observer nodes in HDFS by adding a last state ID returned by the active, fast journal tailing (pull) on standbys, and a metadata msync API plus automatic periodic syncs. Deployed on LinkedIn clusters (≈650 PB total, up to 800M objects per cluster, ~70k RPCs/s) and upstreamed to open source. Result: read latency drops from ~50 ms to ~1 ms, writes stay ~17 ms, and most read QPS shifts off the active.
AI-written summary · about
chapters (22)
- 0:00Introduction
- 1:09Agenda
- 1:54Active Standby
- 3:41Distributed File System
- 6:14Data Node Layer
- 7:35Pop Quiz
- 8:36Hadoop Infrastructure
- 9:31Motivation
- 10:15Standby Node
- 11:11Consistency Model
- 12:23Read Your Own Rights
- 12:59ThirdParty Communication
- 13:33ThirdParty Communication Examples
- 15:10LastState ID
- 17:15Fast Journaling
- 18:10AMSync
- 21:33Always Active
- 23:21New Architecture
- 23:56Magical Performance
- 24:56Rolling Upgrade Procedure
- 25:39Intermediate Results
- 27:41implicit Fsync
related talks
20mFine Grained Replicated State Machines for a Cluster Storage System
24mStronger Semantics for Low-Latency Geo-Replicated Storage
16mHolistic and Automated Task Scheduling for Distributed LSM-tree-based Storage
25mCloud native data replication strategies
23mRobustness in the Salus Scalable Block Store
20mFrom timeout to sub-second: solving scale dependent deadlocks in distributed systems
27mFrom Thundering Herd to Zero Outages: Building Reliable Inventory Sync
22mJoining Dozens of Data Streams in Distributed Stream Processing Systems