How to build a planet-scale data replication...
summary
Google builds a planet-scale replication service on Colossus with a Colossus curators control plane and a separate D data plane plus custodians for recovery; it copies at aggregate rates measured in terabytes per second by sharding files into many 4–8 MiB pieces and split file into millions of elements to copy in parallel, orchestrating source/destination resources in a distributed way. Design choices: Bigtable metadata, erasure coding (cold write amplification <2, hot ~2–3), millions of RPCs per server, and checksums at multiple layers for silent bit flips.
AI-written summary · about
chapters (5)
- 0:00Introduction and team
- 6:58Storage infrastructure overview
- 17:24Cross-cluster data replication
- 28:02Handling massive scale challenges
- 34:06Conclusion and Q&A
related talks
31mGoogle Cloud Bigtable: petabyte-scale NoSQL database for scalable computation
48mGet to know Google Cloud SQL
31mReliability and Resilience in the Cloud
30mBuilding a Distributed Build System at Google Scale
30mBuilding distributed systems is hard - case study (...)
47mWeb-Scale IT Engineering.
34mArchitecting Solr indexing pipelines in Google Cloud Platform
25mCloud native data replication strategies