Data Validation for Feature pipelines: Using Great Expectations and Hopsworks
summary
Feature pipelines validate at write time into a feature store by attaching an expectation suite to a Hopsworks feature group so data is checked before ingestion. Implementation uses Great Expectations (profiles proposing checks, detailed validation reports), compares alternatives like Pydantic and Pandera, and supports strict or “ingest anyway” modes, alerts to Slack/PagerDuty, and history tracking. Demo shows profiling suggesting 56 expectations with six failing and a pipeline that runs seven expectations and returns detailed failure rows for debugging.
AI-written summary · about
chapters (10)
- 0:00Data validation for feature pipelines
- 2:56Duplicated validation logic problems
- 5:05Feature store simplifies feature engineering
- 9:59Evolving data in feature stores
- 10:22Data validation libraries comparison
- 14:46Great Expectations core concepts
- 16:31Integrating Great Expectations with feature store
- 19:25Live demo: Great Expectations with Hopsworks
- 23:44Handling validation failures and alerts
- 25:53Configuring expectations and memory
related talks
23mHopsworks Feature Store after 4 years: Lessons learned and what's next
05mData Quality and Validation in ML Pipelines
26mLarge Scale Data Validation (with Spark and Dask)
01h 28mFeature Engineering Made Simple
20mTime Travel and Provenance for Machine Learning Pipelines
28mData VS Engineers: a long lasting story
34mPanel 02 - Data Quality & Feature Stores
20mDevMLOps at Scale