Petabytes in YDB over HDD in Yandex.Metrica Processing
summary
A case study of building a petabyte-scale YDB (Yandex Database) storage layer on HDDs for Yandex.Metrica’s mobile analytics system, describing architectural choices and implementation details. It covers challenges of using HDDs at that scale and the specific solutions used to make YDB work reliably and performantly for large analytics workloads.
AI-written summary · about
chapters (12)
- 0:00Yandex.Metrica introduction
- 3:03Data processing in Metrica
- 7:26New conversion tasks
- 9:35Storage technology choice
- 13:06Conversion processing scheme
- 16:18Data storage optimization
- 18:27HDD read problems
- 21:25Performance solution
- 23:10Adding buffer to scheme
- 26:34Partition and compaction optimization
- 27:59HDD cost benefits
- 32:46Q&A
related talks
26mImplementing YDB CDC: The Yandex Monitoring Case
51mYandex Database: Distributed Queries in the Clouds
48mYTsaurus — The Future of DWH, Already Here in Yandex Market
40mStorage Architecture for Yandex.Direct Advertising Objects
40mDatabases in the Cloud
37mDeduplication of 5 Million Events per Second on YDB in AppMetrica
30mFrom monolith to petabyte scale: Built a cloud-native, AI-ready data layer
55mYandex.Metrica and a Non‑standard ClickHouse