Related papers: A tutorial on automatic differentiation with compl…
Based on a class of associative algebras with zero-divisors which are called real-like algebras by us, we introduce a way of defining automatic differentiation and present different ways of doing automatic differentiation to compute the…
Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold R, however, is vital for achieving high accuracy under DP. We…
Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic…
This contribution proposes a new formulation to efficiently compute directional derivatives of order one to fourth. The formulation is based on automatic differentiation implemented with dual numbers. Directional derivatives are particular…
This paper presents a novel algorithm to obtain the closed-form anti-derivative of a function using Deep Neural Network architecture. In the past, mathematicians have developed several numerical techniques to approximate the values of…
Shape optimization approaches to inverse design offer low-dimensional, physically-guided parameterizations of structures by representing them as combinations of shape primitives. However, on discretized rectilinear simulation grids,…
This article aims to demonstrate and discuss the applications of automatic differentiation (AD) for finding derivatives in PDE-constrained optimization problems and Jacobians in non-linear finite element analysis. The main idea is to…
In this pedagogical article, we present a simple direct matrix method for analytically computing the Jacobian of nonlinear algebraic equations that arise from the discretization of nonlinear integro-differential equations. The method is…
Automatic differentiation plays a prominent role in scientific computing and in modern machine learning, often in the context of powerful programming systems. The relation of the various embodiments of automatic differentiation to the…
We present two analytical formulae for estimating the sensitivity -- namely, the gradient or Jacobian -- at given realizations of an arbitrary-dimensional random vector with respect to its distributional parameters. The first formula…
Algorithmic differentiation (AD) tools allow to obtain gradient information of a continuously differentiable objective function in a computationally cheap way using the so-called backward mode. It is common practice to use the same tools…
Particle filters are not compatible with automatic differentiation due to the presence of discrete resampling steps. While known estimators for the score function, based on Fisher's identity, can be computed using particle filters, up to…
Optimization of beamlines and lattices is a common problem in accelerator physics, which is usually solved with semi-analytical methods and numerical optimization routines. However, these are usually of the gradient-free or…
Many algorithms for control, optimization and estimation in robotics depend on derivatives of the underlying system dynamics, e.g. to compute linearizations, sensitivities or gradient directions. However, we show that when dealing with…
Machine learning and neural network models in particular have been improving the state of the art performance on many artificial intelligence related tasks. Neural network models are typically implemented using frameworks that perform…
In this paper we introduce DiffSharp, an automatic differentiation (AD) library designed with machine learning in mind. AD is a family of techniques that evaluate derivatives at machine precision with only a small constant factor of…
Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas.…
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs. The overhead complexity of the…
Automatic differentiation, as implemented today, does not have a simple mathematical model adapted to the needs of modern machine learning. In this work we articulate the relationships between differentiation of programs as implemented in…
We describe efficient differentiation methods for computing Jacobians and gradients of a large class of matrix functions including the matrix logarithm $\log(A)$ and $p$-th roots $A^{\frac{1}{p}}$. We exploit contour integrals and conformal…