Anatomy of a production ML feature engineering platform
summary
Claim: feature engineering becomes its own production problem once features explode and run daily; Scribble builds a platform (Enrich) that tracks git commits, enforces validation, and exposes lineage so numbers can be defended. Concrete evidence: an IoT pipeline of 6,000 lines of pandas runs 24–48 hours, retail needed tagging for 15,000 SKUs, teams generate 40,000 feature sets over time, and only about 1 in 50 models reaches production, so reuse, specs (Feast), and automation save major cost.
AI-written summary · about
chapters (23)
- 0:00Intro
- 2:40Michelangelo
- 3:53What did they build right
- 7:42What can we learn
- 10:32What is feature engineering
- 12:48Two big drivers
- 13:39A lot space
- 14:31Three subproblems
- 18:30Most organizations are already burdened
- 19:40What do we want
- 20:34What is missing
- 22:36Specification language
- 24:09Gojek
- 24:47tacit knowledge
- 26:05planning
- 26:25reconciling data
- 28:36validation
- 32:29first world version
- 32:57input correctness
- 34:36validate data
- 35:13data science
- 36:59economics
- 38:26questions to ask
related talks
30mHow Feature Stores Enable Operational ML
21mTime for quality ML from event-based data
01h 28mFeature Engineering Made Simple
45mBuilding a Feature Platform to Scale Machine Learning
26mReducing the cost of production AI
09mPlatform Engineering for Data Science at Scale
36mBuild prod ML systems with feature /training /inference pipelines
29mDemystifying Machine Learning in Production: Reasoning about a Large-Scale ML Platform