Semblance: Feature Generation In Real-Time and Batch Without Time-Travel
summary
Semblance implements an FRP-style API with events and features so feature values derive only from past events, eliminating no time travel mistakes; a single codebase compiles to Scalding Spark Flink Storm or custom real-time compilers. Production metrics: deployment velocity improves from months to ~4 weeks (≈12x), feature match rate rises from 85% to 98% match rate, model AUC/throughput gains ~35%, largest feature uses ~90 TB with 22B keys at 10k events/s and 1.2M ops, p50 <30ms p99 <200ms.
AI-written summary · about
chapters (15)
- 0:00Introduction and speaker intro
- 1:07Problems with old feature generation
- 2:31Time travel in machine learning
- 3:56Mismatched features for training and scoring
- 5:20Functional reactive programming concepts
- 6:47Events and features in Semblance
- 8:09Feature operators and types
- 9:34Generating Semblance features code example
- 11:19Lookup joins in Semblance
- 13:53System architecture overview
- 14:57Decoupling business logic and backends
- 16:23Semblance v2 and performance gains
- 17:31Production system metrics
- 17:52Future work and feature complexity
- 18:56Future plans and acknowledgments
related talks
40mCreating and operating ML models from event based data using feature stores
21mTime for quality ML from event-based data
17mCreating & operating ML models from event-based data using feature stor...
01h 28mFeature Engineering Made Simple
15mOn-demand low-latency feature generation at Hyperconnect
20mUsing Flink for Versatile Feature Engineering Pipelines
31mBuilding an open-source ML feature store with Apache Flink
20mBuilding FeatureFlow, Criteo’s feature data generation platform