How to quickly build Data Pipelines for Data Scientists
summary
Claim: build runnable data pipelines fast using pyspark and Delta Lake with notebook-driven flows. Implementation details: use streaming reads with triggerOnce and checkpointing to make loads idempotent, use a merge upsert for full-load diffs, and keep clear data-engineer / data-scientist boundaries. Demo reads 180k wine reviews, appends ~1,000 new rows and updates ~100 rows, shows version history and time-travel queries, notes delta can be read without Spark but local disk versioning and scalability require planning.
AI-written summary · about
chapters (10)
- 0:00Introduction
- 1:58Data Pipelines
- 7:32Tip 1: Define a Clear Split Between Data Engineering and Data Science
- 8:58Tip 2: For the Data Pipeline Part Use Notebooks in Flows
- 10:34Tip 3: Have a Sit-Down With Your Team and Decide on Standards
- 11:14Tip 4: All Data You Use Needs to Have a Source
- 12:36Tip 5: Prepare for Changes
- 13:36Delta Lake
- 16:38Demo
- 26:50Conclusion
related talks
33mYou Don’t Need Spark for That: Pythonic Data Lakehouse Workflows
29mData Acquisition: Overcoming Challenges
01h 09mNotebook to Pipeline: Hands-On Data Engineering w Python
42mWeb Development To Big Data. A Journey
01h 28mBuilding Reliable, Open Lakehouses with Delta Lake
35mData Processing Pipelines for Small Big Data
19mDemystifying Modern Data Pipeline Architecture
01h 35mDeep Dive into the Big Data Landscape