Implementing YDB CDC: The Yandex Monitoring Case
summary
Describes deploying YDB Change Data Capture (CDC) in Yandex Monitoring to stream and replicate metric writes. Covers using YDB CDC to handle very high write throughput (about 700 million metric points per second) for internal Yandex services and external users in Yandex.Cloud, and the engineering steps and integrations required to process that load.
AI-written summary · about
chapters (19)
- 0:00Introduction
- 1:12Cache invalidation issue
- 3:00Table re-read problems
- 4:49CDC technologies theory
- 6:59YDB CDC advantages
- 8:45YDB CDC workflow
- 10:14Virtual timestamps
- 11:43Referential integrity
- 12:47CDC limits and loads
- 13:52Production issues
- 14:40Performance analysis
- 15:21Optimization resources
- 15:42Bulk table listing
- 16:24Load balancer issues
- 17:49Data reading methods
- 18:36Testing and sync
- 19:18Monitoring and metrics
- 20:02Implementation results
- 21:28Summary and plans
related talks
42mPetabytes in YDB over HDD in Yandex.Metrica Processing
22mChange Data Capture: Exploring YugabyteDB’s Modern Design
51mYandex Database: Distributed Queries in the Clouds
37mDeduplication of 5 Million Events per Second on YDB in AppMetrica
47mState machines for product data from YDB and queues
20mChange Data Capture in Distributed Systems
39mDeep Dive on Using YugabyteDB's Change Data Capture Connector
24mCDC: From Zero to Hero