Non-local compiler transformations in the presence of dynamic dispatch
summary
Describes a concrete compiler approach: implement reverse-mode AD as a functor that returns the primal value plus an opaque closure residual (source-to-source), use ChainRules-style R-rules, and exploit an optical constructions isomorphism so higher-order derivatives reuse the same representation. Handles fully dynamic Julia (eval, exceptions), targets 4–5th derivatives in practice, and relies on opaque-closure semantics (typed by signature, ignore later method-table redefinitions) so the optimizer can collapse and inline the huge backward closures for production performance.
AI-written summary · about
chapters (24)
- 0:00Introduction
- 0:42AD System Requirements
- 1:42Generality in AD Systems
- 4:23Flexibility and Custom Rules
- 5:34Performance Expectations
- 6:12Introducing Defrator.jl
- 8:09Differential Geometry Basics
- 10:37Tangent Bundles and Fields
- 12:43Pushforwards in Differentiation
- 15:39Pushforward in 2D
- 17:11Gradients and Vector Fields
- 20:08Cotangent Vectors and Pullbacks
- 22:37Composition of Functions
- 24:24Reverse Mode AD Abstraction
- 29:47Optics in Category Theory
- 32:25Reverse Mode and Optics
- 33:47Zygote and Compiler AD
- 35:27Higher-Order Optics
- 37:43Julia's Performance Model
- 40:10Opaque Closures
- 41:43Implementing Gradients
- 43:53Optic Isomorphisms
- 45:28System Summary and Future
- 46:30Q&A Session
related talks
29mStaging Automatic Differentiation with Fusion
23mContextual source code AD transformations for sum types
24mEfficient Late Binding of Dynamic Function Compositions
23mEfficient Dual-Numbers Reverse AD via Well-Known Program Transformations
25mYou Only Linearize Once: Tangents Transpose to Gradients
29mPartial Evaluation of Automatic Differentiation for Differential-Algebraic Equati...
29mParallelism-preserving automatic differentiation for second-order array languages
05mPartial Evaluation of Forward-Mode Automatic Differentiation