Related papers: Automatic Differentiation: Inverse Accumulation Mo…
We present two analytical formulae for estimating the sensitivity -- namely, the gradient or Jacobian -- at given realizations of an arbitrary-dimensional random vector with respect to its distributional parameters. The first formula…
Solving complex optimization problems in engineering and the physical sciences requires repetitive computation of multi-dimensional function derivatives. Commonly, this requires computationally-demanding numerical differentiation such as…
In appropriate frameworks, automatic differentiation is transparent to the user at the cost of being a significant computational burden when the number of operations is large. For iterative algorithms, implicit differentiation alleviates…
Considered herein is a modified Newton method for the numerical solution of nonlinear equations where the Jacobian is approximated using a complex-step derivative approximation. We show that this method converges for sufficiently small…
In this pedagogical article, we present a simple direct matrix method for analytically computing the Jacobian of nonlinear algebraic equations that arise from the discretization of nonlinear integro-differential equations. The method is…
The direct or algorithmic approach for the Jacobian problem, consisting of the direct construction of the inverse polynomials is proposed. The so called principle and derived Jacobi conditions are proposed and discussed. The algorithmic…
We give a simple example of non-uniqueness in the inverse scattering for Jacobi matrices: roughly speaking $S$-matrix is analytic. Then, multiplying a reflection coefficient by an inner function, we repair this matrix in such a way that it…
Jacobian-vector products (JVPs) form the backbone of many recent developments in Deep Networks (DNs), with applications including faster constrained optimization, regularization with generalization guarantees, and adversarial example…
While backpropagation--reverse-mode automatic differentiation--has been extraordinarily successful in deep learning, it requires two passes (forward and backward) through the neural network and the storage of intermediate activations.…
We establish an invertibility criterion for free polynomials and free functions evaluated on some tuples of matrices. We show that if the derivative is nonsingular on some domain closed with respect to direct sums and similarity, the…
Some general problems of Jacobian computations in non-full rank matrices are discussed in this work. In particular, the Jacobian of the Moore-Penrose inverse derived via matrix differential calculus is revisited. Then the Jacobian in the…
We introduce a new set of algorithms to compute Jacobi matrices associated with measures generated by infinite systems of iterated functions. We demonstrate their relevance in the study of theoretical problems, such as the continuity of…
Multi-wave inverse problems are indirect imaging methods using the interaction of two different imaging modalities. One brings spatial accuracy, and the other contrast sensitivity. The inversion method typically involve two steps. The first…
The efficient computation of Jacobians represents a fundamental challenge in computational science and engineering. Large-scale modular numerical simulation programs can be regarded as sequences of evaluations of in our case differentiable…
Variational inequalities represent a broad class of problems, including minimization and min-max problems, commonly found in machine learning. Existing second-order and high-order methods for variational inequalities require precise…
Adjoint methods have gained popularity in recent years for driving adaptation procedures which aim to reduce error in solution functionals. While adjoint methods have been proven effective for functional-based adaptation, the practical…
At the heart of Newton based optimization methods is a sequence of symmetric linear systems. Each consecutive system in this sequence is similar to the next, so solving them separately is a waste of computational effort. Here we describe…
We describe a three precision variant of Newton's method for nonlinear equations. We evaluate the nonlinear residual in double precision, store the Jacobian matrix in single precision, and solve the equation for the Newton step with…
We present an algorithmic equivalent statement to the Jacobian conjecture. Given a polynomial map F on an affine space of dimension n, our algorithm constructs n sequences of polynomials such that F is invertible if and only if the zero…
In distributed optimization and distributed numerical linear algebra, we often encounter an inversion bias: if we want to compute a quantity that depends on the inverse of a sum of distributed matrices, then the sum of the inverses does not…