When the data levee breaks: Managing cognitive load in data-intensive projects
summary
Modern data projects pile up pipelines until teams drown; this report shows concrete fixes: use data mesh plus stream-aligned teams and a platform to enforce ownership; capture schemas and freshness in a data catalog lineage to stop repeated production breaks (one real bug was an ID switching from integer to string); standardize metric scaffolding with a DSL and apply clean architecture folder rules so dozens of metrics scale and on-call time drops.
AI-written summary · about
chapters (18)
- 0:00Intro
- 1:19What are we going to talk about?
- 3:55Data engineering projects seem harder, no?
- 7:03Grape farmers cooperative
- 10:32Common symptoms of your team being overloaded with information
- 11:40Data mesh is a decentralized socio-technical approach to manage analytical data at scale
- 12:25What is the advice from Team Topologies?
- 13:01How does this help reduce cognitive load?
- 14:41How would we manage knowledge in a project? We are going to talk about all the documents you need!
- 14:56Use cases What documentation do we need?
- 16:47What if a pipeline failed because of a type error? One source of truth when we document data contracts
- 18:20Automate schema tracking! Example dataset from a Data Catalog
- 21:26Automate keeping track of the data flow Data Catalogs lineage feature is useful
- 22:12Should metrics be developed uniformly? Map out patterns and start creating scaffolding around them
- 23:54How do I develop a new metric? Visualize data transformations
- 26:42How do I make sense of the code? Clean architecture
- 28:51Why should we follow Technical team can understand in a quick look what is the data project calculating this structure?
- 30:17How to reduce cognitive load?
related talks
30mDesigning Resilient Data Pipelines
42mBringing Software Engineering Rigour to Data, DataEngBytes 2022
52mScaling Data Teams: Three Pitfalls and How To Avoid Them
20mScale-up your job satisfaction, not your software
38mKeeping pace with data infrastructure so it doesn't outrun you
31mData Unleashed: A Developer's Perspective on Navigating the Architecture Maze
41mModern Data Architectures – What They Are and How to Implement?
37mThe perils of building a democratic data platform