Related papers: A tutorial on automatic differentiation with compl…
We aim at computing the derivative of the solution to a parametric optimization problem with respect to the involved parameters. For a class broader than that of strongly convex functions, this can be achieved by automatic differentiation…
Algorithmic differentiation (AD) allows exact computation of derivatives given only an implementation of an objective function. Although many AD tools are available, a proper and efficient implementation of AD methods is not…
Automatic differentiation (AD) has been a topic of interest for researchers in many disciplines, with increased popularity since its application to machine learning and neural networks. Although many researchers appreciate and know how to…
A characteristic feature of differential-algebraic equations is that one needs to find derivatives of some of their equations with respect to time, as part of so called index reduction or regularisation, to prepare them for numerical…
This course, intended for undergraduates familiar with elementary calculus and linear algebra, introduces the extension of differential calculus to functions on more general vector spaces, such as functions that take as input a matrix and…
Automatic differentiation (AD) is an ensemble of techniques that allow to evaluate accurate numerical derivatives of a mathematical function expressed in a computer programming language. In this paper we use AD for stating and solving solid…
In min-min optimization or max-min optimization, one has to compute the gradient of a function defined as a minimum. In most cases, the minimum has no closed-form, and an approximation is obtained via an iterative algorithm. There are two…
In this paper we take a look at Automatic Differentiation through the eyes of Tensor and Operational Calculus. This work is best consumed as supplementary material for learning tensor and operational calculus by those already familiar with…
Automatic differentiation frameworks are optimized for exactly one thing: computing the average mini-batch gradient. Yet, other quantities such as the variance of the mini-batch gradients or many approximations to the Hessian can, in…
Forward Automatic Differentiation (AD) is a technique for augmenting programs to compute derivatives. The essence of Forward AD is to attach perturbations to each number, and propagate these through the computation. When derivatives are…
Fractional-order differentiation has many characteristics different from integer-order differentiation. These characteristics can be applied to the optimization algorithms of artificial neural networks to obtain better results. However, due…
While backpropagation--reverse-mode automatic differentiation--has been extraordinarily successful in deep learning, it requires two passes (forward and backward) through the neural network and the storage of intermediate activations.…
Accurate derivatives are important for efficiently locally traversing and converging in quantum optimization landscapes. By deriving analytically exact control derivatives (gradient and Hessian) for unitary control tasks, we show here that…
A unified explicit form for difference formulas to approximate the fractional and classical derivatives is presented. The formula gives finite difference approximations for any classical derivatives with a desired order of accuracy at nodal…
Most nonlinear partial differential equation (PDE) solvers require the Jacobian matrix associated to the differential operator. In PETSc, this is typically achieved by either an analytic derivation or numerical approximation method such as…
Stencil loops are a common motif in computations including convolutional neural networks, structured-mesh solvers for partial differential equations, and image processing. Stencil loops are easy to parallelise, and their fast execution is…
The problem of exactly differentiating a signal with bounded second derivative is considered. A class of differentiators is proposed, which converge to the derivative of such a signal within a fixed, i.e., a finite and uniformly bounded…
We present a differential-geometric formulation of automatic differentiation (AD) based on jet functors and Weil algebras. In this framework, forward- and reverse-mode differentiation arise naturally as pushforward and cotangent pullback,…
This document presents a new C++ Automatic Differentiation (AD) tool, AD-HOC (Automatic Differentiation for High-Order Calculations). This tool aims to have the following features: -Calculation of user specified derivatives of arbitrary…
First-order automatic differentiation is a ubiquitous tool across statistics, machine learning, and computer science. Higher-order implementations of automatic differentiation, however, have yet to realize the same utility. In this paper I…