Scaling Feature Engineering with Dagger

on YouTube ↗18m

" summary

Dagger is a cloud-native framework built on Apache Flink that runs feature engineering as SQL first pipelines with optional Python or Java UDFs and GitOps YAML deployment, powering 300+ jobs written by 50+ data scientists and processing ~10+ TB daily. It implements hybrid source backfills that switch from Parquet to Kafka automatically, stream enrichment calls to external stores, configurable data masking, and runs on managed Flink or Kubernetes clusters for elastic production execution.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 0:58Need of Feature Engineering
3. 1:31Feature Engineering Challenges
4. 4:49How Dagger Works
5. 6:57Key Features
6. 7:57Job Service
7. 8:27Git Ops
8. 11:18Hybrid Data Source
9. 13:26Stream Enrichment

" related talks

• 36mDeclarative Feature EngineeringMiguel Leite, Vitalii Zhebrakovskyi · PyData Amsterdam 2025
• 15mFeature Stores: Powering Machine Learning Infrastructure with Apache FlinkSimba Khadder · Flink Forward Global Virtual 2020
• 20mUsing Flink for Versatile Feature Engineering PipelinesMichal Ciesielczyk · Flink Forward Global Virtual 2020
• 21mReal-time Stream Analytics and Scoring Using Apache Flink, Druid & CassandraCiesielczyk, Zontek · Flink Forward Europe 2019
• 28mData VS Engineers: a long lasting storyAndrea Purgato, Federica Previ · PyCon Italia 2024
• 20mBuilding FeatureFlow, Criteo’s feature data generation platformPiyush Narang · Flink Forward Virtual 2020
• 31mBuilding an open-source ML feature store with Apache FlinkRoman Grebennikov · Flink Forward Global 2021
• 45mBuilding a Feature Platform to Scale Machine LearningWill Pienaar · ODSC India 2018