Hopsworks Feature Store after 4 years: Lessons learned and what's next
summary
After four years building Hopsworks the team claims to have solved the core “data write” and “data access” problems by combining an offline/online store, streaming writes to the online store, and a python-first API that layers on top of Spark when needed. Concrete pieces: feature groups for write-time validation (Great Expectations runs before ingest), feature views for point-in-time joins, TFRecord export and model-specific transformations applied at retrieval, and a KServe integration for serving. A free hosted trial (app.ei) demonstrates the end-to-end pipeline working in practice.
AI-written summary · about
chapters (7)
- 0:00Operationalizing ML
- 1:47Data engineering challenges
- 4:51Python-first feature store
- 8:09Solving data write problems
- 12:34Accessing features and views
- 17:29Real-time prediction services
- 19:26Q&A and closing
related talks
32mA feature Store for Machine Learning
30mHow Feature Stores Enable Operational ML
20mNavigating the Skies of Data: A Flight Through Feature Stores
33mEvolution and Scaling of Feature Store at Uber
15mFeature Stores: Powering Machine Learning Infrastructure with Apache Flink
33mFeathr, an Open Source, Battletested, Scalable and Enterprise-Grade Feature Store
28mData VS Engineers: a long lasting story
28mData Validation for Feature pipelines: Using Great Expectations and Hopsworks