How to build a data pipeline without data
summary
Synthetic data replaces absent production rows by generating realistic records with Python and writing SQL migrations; the demo builds a Postgres schema via SQLAlchemy, produces data with Faker, and stages repeatable loads with Flyway. It actually creates ~1,000 customers plus 60 products, 1,000 devices and ~5,000 transactions, exports INSERT scripts, applies them, and verifies rows exist. Pros: speeds development, enables edge‑case tests; cons: may miss real‑data nuance and requires maintenance.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:12What are data pipelines
- 3:43Synthetic data with Faker
- 10:30Realism limits in synthetic data
- 14:46Creating synthetic data objects
- 17:38Writing data with Flyway
- 22:18Flyway migrations
- 25:29Pros and cons of synthetic data
- 26:54Best practices for synthetic data
- 27:15Summary and Q&A
related talks
27mHow To Generate Test Data for Your Database Project
01h 10mReal World Iterative Database Development
35mAvoiding Bad Database Surprises: Simulation and Scalability
47mEfficiency up, failed releases down; compliant production data for your test and dev environments
32m7 steps to build-your-own data pipeline - for day 1 of your startup
32mScalable Python and SQL Data Engineering without Migraines
41m7 steps to build your own data pipeline
17mOpen Source for Data Quality