The perils of building a democratic data platform
summary
Company runs a CDC ingestion pipeline on Kafka and a centralized processing layer that handles 80000 unique datasets and over several petabytes read per day and hundreds of terabytes written daily. Built a scheduler, serving layer, and metadata system but finds custom metadata brittle so adopts open lineage instead. Growth exposes failures: hidden transitive dependencies, runaway costs, unclear ownership. Practical fixes: enforce limit transitive dependencies to 10, treat datasets as data products with cost attribution and SLAs.
AI-written summary · about
chapters (8)
- 0:00Intro
- 0:42About New Bank
- 6:16How we evolved the platform
- 9:04When to adopt complex technology
- 14:37Misleading consolidation for centralization
- 22:40Platformcentric perspective
- 29:58Overlooking the changes required
- 31:43Questions
related talks
31mWhen the data levee breaks: Managing cognitive load in data-intensive projects
46mJourney of metadata
33mAccidental DataOps: How we matured our organization with data, and how you can too!
27mRecipe for Building a Discoverable and Governed Data Platform.
34mTales of Data Architecture Evolution
34mData at scale: How to deal with small challenges when they become massive
17mDemocratising data analytics
35mBuilding a data analytics platform on strong opinions held loosely