Understanding Automatic Differentiation Pitfalls
Abstract
Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives computed by AD could be interpreted as incorrect. These pitfalls occur systematically across tools and approaches. In this paper we broadly categorize problematic usages of AD and illustrate each category with examples such as chaos, time-averaged oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers. We also review debugging techniques and their effectiveness in these situations. With this article we hope to help readers avoid unexpected behavior, detect problems more easily when they occur, and have more realistic expectations from AD tools.
Cite
@article{arxiv.2305.07546,
title = {Understanding Automatic Differentiation Pitfalls},
author = {Jan Hückelheim and Harshitha Menon and William Moses and Bruce Christianson and Paul Hovland and Laurent Hascoët},
journal= {arXiv preprint arXiv:2305.07546},
year = {2023}
}