Reverse Automatic Differentiation for Accelerate (Extended Abstract)
summary
Implementation of reverse AD for Accelerate as a compiler transformation that turns array indexing into a dual array indexing → permute (scatter) operation, with support for map, fold/reduce and simple scans but no general scan derivative or sequential loops. Benchmarks use ADBench GMM and bundle-adjustment on CPU, comparing to TensorFlow, PyTorch, manual C++ and F#; GMM runs out of memory in this prototype, BA is reasonable single-threaded and better multi-threaded. Key takeaway: indexing dualizes to a scatter and causes significant memory pressure.
AI-written summary · about
chapters (5)
related talks
29mPartial Evaluation of Automatic Differentiation for Differential-Algebraic Equati...
29mAutomatic Differentiation via Effects and Handlers in OCaml
05mPartial Evaluation of Forward-Mode Automatic Differentiation
23mEfficient Dual-Numbers Reverse AD via Well-Known Program Transformations
29mStaging Automatic Differentiation with Fusion
36mProgramming Languages for Automatic Differentiation: What Now?
29mAutomatic differentiation, a tale of two languages
39mDifferentiable Functional Programming