7 steps to build-your-own data pipeline - for day 1 of your startup
summary
A cost‑sensitive startup pipeline that aims for immediate, row‑level access builds a thick client called squirrel then a file injector badger in Go and uses Redshift + S3 + NSQ for storage and transport, starting on a DC2 large and a 120‑day moving window. It ingests 20 billion events per day (~800 GB uncompressed, 200k avg / 350k peak eps), tunes COPY vs INSERT, shards hot tables, and migrates slowly to v2 over months — a working, scalable pipeline that keeps ops light.
AI-written summary · about
chapters (11)
- 0:00Intro
- 0:30Who we are
- 2:10What we wanted on Day 1
- 5:30Steps
- 7:05Understand - Requirements and Constraints
- 10:36Think Generic - Data schema design
- 14:11Produce data well!
- 18:01#14: Design "v1.0" of Data Pipeline
- 22:07Tune & Repeat
- 24:00#7: Upgrade to v2.0 of Data Pipeline
- 24:38Current Scale
related talks
41m7 steps to build your own data pipeline
45mResilient Predictive Data Pipelines
32mDiscover the Modern Way to Build Real-Time Analytics Solutions at Any Scale
21mDesigning Modular Data Pipeline
10mBuilding High-Performance Financial Data Pipelines
34mData Engineering Architecture
17mOptimizing Data-Driven Decisions
22mPractical approach to building real-time analytics for cybersecurity applications