Peachbox: Agile and Accessible Big ETL Framework
summary
Peachbox claims to be a Python Big ETL framework that implements the Lambda Architecture end to end; the demo shows a working pipeline that ingests Amazon movie reviews, stores a fact‑based master dataset, builds hourly batch aggregates and a streaming speed layer, and merges results for queries. Actual stack components are PySpark with Docker + Kafka + Cassandra; the talk shows code for model classes, pipeline validators, a scheduler and live writes to Cassandra, notes eventual consistency and rough edges, and marks the project as usable but not production hardened.
AI-written summary · about
chapters (23)
- 0:00Peachbox introduction
- 2:14Particle physics to real-time bidding
- 6:16Lambda Architecture explained
- 8:52Peachbox framework
- 12:06Peachbox ecosystem
- 13:30Lambda components and data flow
- 17:03Master data set setup
- 20:56Docker environment setup
- 22:28Data model definition
- 23:44Task creation and execution
- 28:00Task scheduling
- 34:10Movie reviews import job
- 37:12Kafka and data ingestion
- 39:41Import job execution
- 41:21Querying with Jupyter
- 42:54Exploring data warehouse
- 47:53Building batch view
- 53:23Querying batch view
- 55:20Real-time views implementation
- 1:01:23Combining batch and real-time
- 1:05:02Lambda Architecture summary
- 1:06:37Peachbox future steps
- 1:09:34Getting started with Peachbox
related talks
39mHow Python Drives Modern Data Engineering
19mDemystifying Modern Data Pipeline Architecture
46mThe Python Data Ecosystem: Navigating a fragmented landscape
44mUsing Python with Hadoop to create production ready big data applications
29mData Acquisition: Overcoming Challenges
35mData Processing Pipelines for Small Big Data
42mDeciphering the way data is tested
44mCreating Python Data Pipelines in the Cloud