Perturbation confusion in forward automatic differentiation of higher-order functions

on YouTube ↗11m

" summary

Fielded forward AD systems produce wrong results for derivatives of higher-order functions because tag creation is tied to invoking the derivative operator, not to each actual derivative calculation; the paper shows a simple curried example where a constructed operator D̂ yields a constant zero instead of the true second derivative. The work diagnoses the bug as perturbation confusion in dual-number/tagging machinery and evaluates two fixes: eta expansion to delay fresh tags and a tag-renaming wrapper, both correct but threatening forward-AD complexity guarantees.

AI-written summary · about

" chapters (10)

1. 0:00Intro
2. 1:09Technical Background and Setup
3. 2:04(1/4) Forward AD-Example
4. 3:12(2/4) Nesting Derivatives - Perturbation Confusion
5. 4:24(3/4) Higher-Order AD-What does it mean?
6. 6:53(4/4) The Amazing Bug - Details Recall
7. 8:38Solution Idea One: Eta Expansion
8. 9:09Solution Idea Two: Tag Substitution
9. 9:34Conclusion
10. 10:48ACKNOWLEDGEMENTS

" related talks

• 29mPartial Evaluation of Automatic Differentiation for Differential-Algebraic Equati...David Broman, Viktor Palmkvist, Oscar Eriksson · GPCE 2023
• 11mAbstract Interpretation for Automatic DifferentiationGagandeep Singh, Jacob Laurel, Sasa Misailovic, Siyuan Brant Qian · LAFI 2024
• 36mProgramming Languages for Automatic Differentiation: What Now?Damiano Mazza · LAFI 2022
• 14mNominal Semantics for First-class Automatic DifferentiationJack Czenszak, Alexander K. Lew · LAFI 2026
• 30mAutomatic Differentiation in PCFDamiano Mazza, Michele Pagani · POPL 2021
• 05mPartial Evaluation of Forward-Mode Automatic DifferentiationDavid Broman, Viktor Palmkvist, Oscar Eriksson · LAFI 2023
• 26mAdding AD to Scheme by Differentiating the InterpreterBarak A. Pearlmutter, Mehrdad Maleki · Scheme 2021
• 29mAutomatic Differentiation via Effects and Handlers in OCamlJesse Sigal · ML 2024