Related papers: Composing Automatic Differentiation with Custom De…
Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning. In this work we articulate the relationships between differentiation of programs as implemented in…
A development of an inverse first-order divided difference operator for functions of several variables is presented. Two generalized derivative-free algorithms builded up from Ostrowski's method for solving systems of nonlinear equations…
There exist many applications where it is necessary to approximate numerically derivatives of a function which is given by a computer procedure. In particular, all the fields of optimization have a special interest in such a kind of…
Automatic differentiation (AD) in reverse mode (RAD) is a central component of deep learning and other uses of large-scale optimization. Commonly used RAD algorithms such as backpropagation, however, are complex and stateful, hindering deep…
This contribution proposes a new formulation to efficiently compute directional derivatives of order one to fourth. The formulation is based on automatic differentiation implemented with dual numbers. Directional derivatives are particular…
Finite differences have been widely used in mathematical theory as well as in scientific and engineering computations. These concepts are constantly mentioned in calculus. Most frequently-used difference formulas provide excellent…
We develop a compositional approach for automatic and symbolic differentiation based on categorical constructions in functional analysis where derivatives are linear functions on abstract vectors rather than being limited to scalars,…
Automatic differentiation is involved for long in applied mathematics as an alternative to finite difference to improve the accuracy of numerical computation of derivatives. Each time a numerical minimization is involved, automatic…
Gradient-based methods for optimisation of objectives in stochastic settings with unknown or intractable dynamics require estimators of derivatives. We derive an objective that, under automatic differentiation, produces low-variance…
Automatic differentiation has become an important tool for optimization problems in computational science, and it has been applied to the Hartree-Fock method. Although the reverse-mode automatic differentiation is more efficient than the…
Many problems in Physics and Chemistry are formulated as the minimization of a functional. Therefore, methods for solving these problems typically require differentiating maps whose input and/or output are functions -- commonly referred to…
In mathematics and computer algebra, automatic differentiation (AD) is a set of techniques to evaluate the derivative of a function specified by a computer program. AD exploits the fact that every computer program, no matter how…
For a real function, automatic differentiation is such a standard algorithm used to efficiently compute its gradient, that it is integrated in various neural network frameworks. However, despite the recent advances in using complex…
A novel class of derivative-free optimization algorithms is developed. The main idea is to utilize certain non-commutative maps in order to approximate the gradient of the objective function. Convergence properties of the novel algorithms…
Automatic Differentiation (AD) is a powerful tool that allows calculating derivatives of implemented algorithms with respect to all of their parameters up to machine precision, without the need to explicitly add any additional functions.…
Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic…
We present a simple functional programming language, called Dual PCF, that implements forward mode automatic differentiation using dual numbers in the framework of exact real number computation. The main new feature of this language is the…
We study the correctness of automatic differentiation (AD) in the context of a higher-order, Turing-complete language (PCF with real numbers), both in forward and reverse mode. Our main result is that, under mild hypotheses on the primitive…
This article aims to demonstrate and discuss the applications of automatic differentiation (AD) for finding derivatives in PDE-constrained optimization problems and Jacobians in non-linear finite element analysis. The main idea is to…
If the result of an expensive computation is invalidated by a small change to the input, the old result should be updated incrementally instead of reexecuting the whole computation. We incrementalize programs through their derivative. A…