Hamilton:Natively bringing SWE best practices to python data transformations
summary
Hamilton is an open‑source Python micro orchestration framework that turns column transforms into declarative functions, builds a DAG from function names and arguments, and runs only required nodes. It integrates with pandas, Polars, NumPy, Pandera checks, and plugins for Dask and Ray, and ties into DBT, FastAPI and Jupyter. At Stitch Fix it moves a team from ~1,000 to 4,000 feature transforms in production and cuts update time from a day to less than two hours.
AI-written summary · about
chapters (5)
- 0:00Introduction and motivation
- 5:58What is Hamilton framework
- 15:30Native software engineering best practices
- 28:54Summary and Hamilton ecosystem
- 30:40Q&A and closing
related talks
21mHamilton: an open source framework for feature transform code in Python
26mIn house to open source Stitching the past to the futu...
01h 01mParallel Python Analyzing Large Data Sets
01h 06mBodo: Automatic HPC Performance and Scaling for Data Processing in Python
27mEngineers to Data-Scientists with Python
25mPerfecting Our Modern Data Stack Recipe: What We Learned On The Journey At Huel
26mLife Beyond Pandas: Workflows with DuckDB, Daft, Polars, and Datafusion
46mThe Python Data Ecosystem: Navigating a fragmented landscape