Designing Modular Data Pipeline
summary
Claims a three-stage pipeline (source → transformer → consumer) that enables swapping components and unit testing, backed by composition and type erasure; lists over 20 OSS libraries used in 80,000 teams; demos file watchers for live-editing JSON and a remote polling variant; describes Oxnap for byte-level record/replay (zero dependencies, default Firebase uploader, 10–30 minute integration) that reproduces ~90% of data bugs and reportedly saved months at the New York Times.
AI-written summary · about
chapters (8)
- 0:00Introduction
- 1:02Modular pipelines with composition
- 4:15Core components
- 7:28Simulating server responses
- 9:35Live file watching
- 13:21Recording user data
- 18:43Replaying recorded data
- 19:48Summary
related talks
19mDemystifying Modern Data Pipeline Architecture
01h 01mCurrent Methods for Building Data Pipelines
32m7 steps to build-your-own data pipeline - for day 1 of your startup
28mYour Pipeline as a Function.
20mThe Pipeline Is Dead
41mModern Data Architectures – What They Are and How to Implement?
41mModular & Reusable: A Recipe to Scaling Data Across a Portfolio
10mBuilding High-Performance Financial Data Pipelines