Real-Time Data Processing for ML Feature Engineering
summary
Meta describes a consolidated real‑time platform that evolved from Stylus, Puma, Turbine into an Extreme platform with an engine that emits machine code or uses a vectorized runtime and a platform layer that autos-scales and load-balances. It implements sliding‑window (leaf/root) mergeable aggregations, non‑mergeable aggregations with a low‑latency user store, and remote map‑join model predictions; internal A/B-style studies report a clear normalized entropy gain for real‑time vs batch features and production deployments run at Meta scale.
AI-written summary · about
chapters (6)
- 0:00Introduction
- 1:57Real-time data for ML
- 3:45Real-time infrastructure evolution
- 7:21Real-time feature use cases
- 12:41Impact on model performance
- 13:44Conclusion & Q&A
related talks
36mHuman-Friendly, Production-Ready Data Science Stack with Metaflow & Kubernetes
12mOpenMLDB: An Open-Source Real-Time Feature Platform
38mReal Time ML Pipelines in Multi-Tenant Environments
17mCreating & operating ML models from event-based data using feature stor...
21mOperationalizing ML Training Infra at Meta Scale
40mCreating and operating ML models from event based data using feature stores
36mEnterprise-grade Full Stack ML Platform
18mThe AI-First Data Infrastructure