SMART DATA PIPELINES FOR AI
summary
Uses FreeMonad DSL to represent pipeline stages as plain data (profiler, transformer, get-metadata) and compiles them to executable recipes that preserve state and audit logs. Builds both a slow training/discovery pipeline and a fast prediction pipeline by omitting profilers, serializes pipeline recipes as JSON to replay past runs, and swaps implicits to run on Spark and in-memory backends. Deployed in production, reduces duplicated pipelines and simplifies cross-cutting concerns.
AI-written summary · about
chapters (9)
- 0:00Introduction
- 3:30Feature Discovery Platform
- 9:33Data Model
- 13:50Compiler
- 16:33Data Profile
- 18:21Prediction Pipeline
- 20:22Implementation
- 21:38Package structure
- 25:29Summary
related talks
43mData Pipelines à la Mode
18mData Preprocessing for ML at Production Scale
38mEnabling Big Data and Machine Learning for the Masses...
41mSpeeding Up Data Pipeline Construction in Apache Spark
18mAutomate the boring ML stuff with pipelines
42mApache Spark Declarative Pipelines in Action
34mData pipelines powered by Open source and fun
31mOne Data Pipeline to Rule Them All