A Type System For Feature Engineering (Extended Abstract)

on YouTube ↗22m

" summary

Built a small compiler for F3-style SQL DSL at Meta and demonstrates a dependent type system (inspired by Martin-Löf) that stores normal forms to make equivalence decidable, using normalization to normal forms and a three-part table type (data schema, normalized filter/join conditions, row/lineage environment). The system tags rows affected by left/right joins, supports about 20 operators, already validates semantic-preserving graph transforms and is being applied to dedupe and index a corpus of 100,000+ programs for querying and reuse.

AI-written summary · about

" chapters (9)

1. 0:00Introduction
2. 0:43Feature engineering
3. 3:11Three DSL operators
4. 5:20Semantic equivalence
5. 6:02Large-scale challenges
6. 8:29Type systems and normalization
7. 10:34Type system for SQL
8. 14:09Types for validation
9. 15:58Q&A

" related talks

• 31mF3: A Compiler For Feature EngineeringSiyu Wang, Weixi Ma, Arnaud Venet, Yafei Yang, Rocky Liu, Junhua Gu, Subbu Subramanian, Daniel P. Friedman · FUNARCH 2024
• 13mDesigning Feature DSLs: Principles and TradeoffsGreg Kuhlmann · Feature Store Summit 2022
• 58mType Theory Behind the Glasgow Haskell Compiler Internals (Part 4)Vitaly Bragilevsky · LambdaConf 2018
• 01h 00mType Theory Behind the Glasgow Haskell Compiler Internals (Part 5)Vitaly Bragilevsky · LambdaConf 2018
• 01h 01mType Theory Behind the Glasgow Haskell Compiler Internals (Part 6)Vitaly Bragilevsky · LambdaConf 2018
• 40mType-Driven Program SynthesisNadia Polikarpova · Strange Loop 2018
• 49mType Theory Behind the Glasgow Haskell Compiler Internals (Part 2)Vitaly Bragilevsky · LambdaConf 2018
• 57mType Theory Behind Glasgow Haskell Compiler Internals (Part 1)Vitaly Bragilevsky · LambdaConf 2018