Related papers: A tutorial on automatic differentiation with compl…
Automatic Differentiation (AD) is instrumental for science and industry. It is a tool to evaluate the derivative of a function specified through a computer program. The range of AD application domain spans from Machine Learning to Robotics…
Diffusive representations of fractional derivatives have proven to be useful tools in the construction of fast and memory efficient numerical methods for solving fractional differential equations. A common challenge in many of the known…
The Cheap Gradient Principle (Griewank 2008) --- the computational cost of computing the gradient of a scalar-valued function is nearly the same (often within a factor of $5$) as that of simply computing the function itself --- is of…
Differentiation is a cornerstone of computing and data analysis in every discipline of science and engineering. Indeed, most fundamental physics laws are expressed as relationships between derivatives in space and time. However, derivatives…
We show that a particular form of target propagation, i.e., relying on learned inverses of each layer, which is differential, i.e., where the target is a small perturbation of the forward propagation, gives rise to an update rule which…
We present a simple yet powerful framework for solving inverse problems by leveraging automatic differentiation. Our method is broadly applicable whenever a smooth cost function can be defined near the true solution, and a numerical…
In the context of the optimization of Deep Neural Networks, we propose to rescale the learning rate using a new technique of automatic differentiation. This technique relies on the computation of the {\em curvature}, a second order…
We discuss the functional lazy techniques in generation and handling of arbitrarily long sequences of derivatives of numerical expressions in one ``variable''; the domain to which the paper belongs is usually nicknamed ``Automatic…
Leibniz's rule for the $n$-th derivative of a product is a very well known and extremely useful formula. In this article, we introduce an analogous explicit formula for the $n$-th derivative of a quotient of two functions. Later, we use…
Performant numerical solving of differential equations is required for large-scale scientific modeling. In this manuscript we focus on two questions: (1) how can researchers empirically verify theoretical advances and consistently compare…
One of the great triumphs in the history of numerical methods was the discovery of the Conjugate Gradient (CG) algorithm. It could solve a symmetric positive-definite system of linear equations of dimension N in exactly N steps. As many…
As more and more multiphysics effects are entering the field of CFD simulations, this raises the question how they can be accurately captured in gradient computations for shape optimization. The latter has been successfully enriched over…
The auto differentiable simulation is a type of simulation that outputs of the simulation include not only the simulation result itself, but also their derivatives with respect to various input parameters. It provides an efficient method to…
There are many approaches for training decision trees. This work introduces a novel gradient-based method for constructing decision trees that optimize arbitrary differentiable loss functions, overcoming the limitations of heuristic…
Differential calculus on discrete spaces is studied in the manner of non-commutative geometry by representing the differential calculus by an operator algebra on a suitable Krein space. The discrete analogue of a (pseudo-)Riemannian metric…
We investigate the automatic differentiation of dominant eigensolver where only a small proportion of eigenvalues and corresponding eigenvectors are obtained. Backpropagation through the dominant eigensolver involves solving certain…
The reverse derivative is a fundamental operation in machine learning and automatic differentiation. This paper gives a direct axiomatization of a category with a reverse derivative operation, in a similar style to that given by Cartesian…
The categorical models of the differential lambda-calculus are additive categories because of the Leibniz rule which requires the summation of two expressions. This means that, as far as the differential lambda-calculus and differential…
Modelers use automatic differentiation (AD) of computation graphs to implement complex Deep Learning models without defining gradient computations. Stochastic AD extends AD to stochastic computation graphs with sampling steps, which arise…
Automatic differentiation has become an important tool for optimization problems in computational science, and it has been applied to the Hartree-Fock method. Although the reverse-mode automatic differentiation is more efficient than the…