7 steps to build your own data pipeline
summary
Covers building a game-company data pipeline end-to-end using S3, Redshift and a custom high-throughput ingestion backend (with a thin client called Badger), addressing scalable ingestion, data design, warehousing and hot-loading into Redshift. Explains why they kept SQL as the query interface, how to support product managers and data scientists with on-demand tables and ad-hoc Redshift clusters, and tuning and bottleneck solutions for scale.
AI-written summary · about
chapters (9)
- 0:00Start
- 6:04What we wanted on Day 1
- 12:53Understand - Requirements and Constraints
- 18:02Think Generic - Data schema design
- 27:42Produce data well!
- 30:34Design "v1.0" of Data Pipeline
- 37:29Tune & Repeat
- 38:15Upgrade to v2.0 of Data Pipeline
- 39:43Current Scale
related talks
32m7 steps to build-your-own data pipeline - for day 1 of your startup
43mData warehousing reinvented for today's needs
45mBuilding an Analytics Platform
30mData Engineering building your BI infrastructure from scratch
42mTransform your data approach: Develop a modern data strategy
31mOne Data Pipeline to Rule Them All
24mA Gentle Introduction to Building Data-Intensive Applications
29mGrowing insights through hybrid data architectures