Featran, Featran77 - Generic Feature Transformer for Data Pipelines
summary
Featran (F77) is a Scala library that builds reusable Map→Reduce→Map feature transformers, computes summaries (min/max, label sets) once and reuses them as JSON settings, and produces dense or sparse vectors feedable into TensorFlow. It uses Algebird semigroups/aggregators, Breeze, Scalacheck and works across Scio, Spark, Scalding, Flink. At Spotify it handles millions of users and petabytes of data, supports Java runtime use, and is proven production-ready though serialization and concurrency required careful fixes.
AI-written summary · about
chapters (11)
- 0:00Introduction and speaker background
- 2:05Feature engineering challenges
- 3:53Basic transformers demo
- 10:21Transformer abstraction with aggregators
- 16:05Feature transformer library demo
- 20:04Feature transformer settings
- 24:20Transformer implementation details
- 27:51Testing and fun facts
- 31:31Performance optimizations
- 32:41Home page ranking use case
- 35:17Q&A session
related talks
30mHow Feature Stores Enable Operational ML
36mDeclarative Feature Engineering
28mData VS Engineers: a long lasting story
33mFeathr, an Open Source, Battletested, Scalable and Enterprise-Grade Feature Store
01h 28mFeature Engineering Made Simple
46mScaling ML Pipelines with Feast, Ray and Kubeflow
20mUsing Flink for Versatile Feature Engineering Pipelines
18mScaling Feature Engineering with Dagger