On Tensors

on YouTube ↗28m

" summary

Current tensor libraries leak low-level flat-memory semantics so reshape and broadcasting rules quietly produce wrong results; the claim is that naming dimensions fixes most surprises and makes shape intent explicit. The talk demonstrates a small DSL that lowers named shapes to library indices, plus a runtime checker called Stanley that flags mismatched shapes, and a lightweight named-dimension library (referred to as TS a lip library) that implements explicit broadcasts and concise transforms. The code examples compile to real NumPy/PyTorch ops and work end to end.

AI-written summary · about

" chapters (21)

1. 0:00Intro
2. 1:10A New Style of Programming
3. 2:07Key Ingredients of Software 2.0 Design
4. 5:05Tensor Representations: Physical Layer
5. 7:36A. Exposure to Low-level Memory Model API tied to physical, indexed layout. No semantic notion of axis
6. 8:53B. Adhoc Broadcast Semantics
7. 10:43C. Hard to read / write Tensor manipulating code
8. 11:54Cryptic Shape Transformations
9. 12:16Tensor Representations: 10ft - 1e10 ft
10. 13:54Proposals for Naming Tensor Dimensions Ideas for array shape typing in Python Annotations #4164
11. 15:12tsalib: A Tensor Shape Annotation Library
12. 16:00Resnet.forward (original)
13. 19:40Named Dimensions -- Grammar of Named Shapes
14. 20:55From Transformer Attention Module (old vs new)
15. 21:38Rewriting BERT with warp
16. 22:39tsalib: Lowering Named Shape Transformations
17. 24:00Two Issues
18. 24:14Tsanley: dynamic shape checking annotation
19. 25:35Tsanley: dynamic shape checking • Interplay of AST parsing, Python tracing (trace, inspect)
20. 25:45Open Source Research @OffNote Labs
21. 26:52Summary: Befriending Tensors • Disconnect between memory model of tensor libraries and developer's semantic model

" related talks

• 27mCatching Tensor Shape Errors without Running Your CodePradeep Kumar Srinivasan · PyCon US 2023
• 01h 02mThe Shapes of Multi-Dimensional ArraysVincent Reverdy · CppCon 2020
• 11mPyTorch 2.0: Dynamic Shapes SupportHorace He · PyTorch Conference 2022
• 33mTeIL: a type-safe imperative Tensor Intermediate LanguageNorman A. Rink, Jeronimo Castrillon · ARRAY 2019
• 08mPyTorch Front-End Features: Named Tensors and Type PromotionGregory Chanan · PyTorch Developer Conference 2019
• 45mTensorly: A Flexible Python Framework for Machine LearningAnima Anandkumar · PyData Los Angeles 2018
• 35mTowards Typesafe Deep Learning in ScalaTongfei Chen · Scale By The Bay 2018
• 04mInterviewTongfei Chen · Scale By The Bay 2018