Josh Wills, Director of Data Science, Cloudera @ MLconf SEA
summary
Argues that feature engineering needs an IDE and shows a working prototype: the Exhibit project layers a Supernova schema (one record per entity with arrays of structs) so users run SQL-in-record to build, save and test features interactively. Demonstrates NFL data and an ETL customer workflow that computes 2500 metrics in a single pass, runs local validation, then exports scaled jobs to Spark/MapReduce — a practical system that already works and is open source.
AI-written summary · about
chapters (14)
- 0:00Intro
- 4:16Definition of a Data Scientist
- 6:50Data Cleansing
- 8:48The key to success
- 10:02Feature engineering
- 10:45Deep learning
- 12:26Is deep learning still necessary
- 13:12What is feature engineering
- 16:17What Josh is interested in
- 16:42Brainwash
- 18:26Operational Analytical Impedance Mismatch
- 19:59Supernova Schema
- 22:06Exhibit
- 23:32Demo
related talks
01h 48mBig Data made easy with a Spark (3.0)
51mSpotlight on Emerging Data Science Worskspaces and Spark as Development Levers
29mData Engineers Shouldn’t Suffer, or Some Things About Big Data Tools for IntelliJ IDEA
28mData VS Engineers: a long lasting story
17mCreating & operating ML models from event-based data using feature stor...
15mSanjeev Mohan, SanjMo | Databricks Data + AI Summit 2023
38mData Secrets from a Platform Engineer
35mPanel 03- How to make the Feature Store become a hidden part of the productionalization of ML models