Streaming Feature Extraction for a ML Pipeline using Memphis.dev and Bytewax.
summary
Shows a working pipeline that uses Memphis.dev as durable event queues and Bytewax for Python stream processing to do online feature extraction instead of heavy batch reprocessing. Demonstrates filtering, enrichment (e.g., ACS ZIP-code data, weather), imputation, windowed aggregation and stateful map updates that emit incremental feature vectors. Implements local state persistence via SQLite-backed state with periodic S3 backup for recovery. Claims reduced fault surface and lower compute cost versus periodic full batch runs.
AI-written summary · about
chapters (7)
- 0:00Introduction and background
- 3:10Why feature extraction matters
- 7:09Online vs offline feature calculations
- 10:27System architecture for online pipelines
- 12:12Memphis and Bytewax overview
- 14:16Combining Memphis with Bytewax
- 16:05Q&A on fault handling
related talks
36mBuild prod ML systems with feature /training /inference pipelines
17mCreating & operating ML models from event-based data using feature stor...
01h 28mFeature Engineering Made Simple
01h 52mDeploying a Fraud Detection Model with a Feature Store
36mDeclarative Feature Engineering
45mAnatomy of a production ML feature engineering platform
40mCreating and operating ML models from event based data using feature stores
30mHow Feature Stores Enable Operational ML