Related papers: Higher-order Reverse Automatic Differentiation wit…
In this work, we concern with the high order numerical methods for coupled forward-backward stochastic differential equations (FBSDEs). Based on the FBSDEs theory, we derive two reference ordinary differential equations (ODEs) from the…
Many developments in Mathematics involve the computation of higher order derivatives of Gaussian density functions. The analysis of univariate Gaussian random variables is a well-established field whereas the analysis of their multivariate…
High-performance Ising machines for solving combinatorial optimization problems have been developed with digital processors implementing heuristic algorithms such as simulated bifurcation (SB). Although Ising machines have been designed for…
The Newton, Gauss--Newton and Levenberg--Marquardt methods all use the first derivative of a vector function (the Jacobian) to minimise its sum of squares. When the Jacobian matrix is ill-conditioned, the function varies much faster in some…
The use of neural networks for solving differential equations is practically difficult due to the exponentially increasing runtime of autodifferentiation when computing high-order derivatives. We propose $n$-TangentProp, the natural…
In this paper, we develop a numerical algorithm for an inverse problem on determining fractional orders of time derivatives simultaneously in a coupled subdiffusion system. Following the theoretical uniqueness, we reformulate the order…
Stochastic second-order methods achieve fast local convergence in strongly convex optimization by using noisy Hessian estimates to precondition the gradient. However, these methods typically reach superlinear convergence only when the…
A typical system of k difference (or differential) equations can be compressed, or folded into a difference (or ordinary differential) equation of order k. Such foldings appear in control theory as the canonical forms of the controllability…
We extend the nonlocal operator method to higher order scheme by using a higher order Taylor series expansion of the unknown field. Such a higher order scheme improves the original nonlocal operator method proposed by the authors in [A…
The topics of source seeking and Newton-based extremum seeking have flourished, independently, but never combined. We present the first Newton-based source seeking algorithm. The algorithm employs forward velocity tuning, as in the very…
Using the notion of conservative gradient, we provide a simple model to estimate the computational costs of the backward and forward modes of algorithmic differentiation for a wide class of nonsmooth programs. The overhead complexity of the…
This article provides an overview of some of the mathematical principles of Automatic Differentiation (AD). In particular, we summarise different descriptions of the Forward Mode of AD, like the matrix-vector product based approach, the…
We develop several new communication-efficient second-order methods for distributed optimization. Our first method, NEWTON-STAR, is a variant of Newton's method from which it inherits its fast local quadratic rate. However, unlike Newton's…
Derivatives of computer graphics, image processing, and deep learning algorithms have tremendous use in guiding parameter space searches, or solving inverse problems. As the algorithms become more sophisticated, we no longer only need to…
In this paper, we introduce a new higher-order directional derivative and higher-order subdifferential of Hadamard type of a given proper extended real function. This derivative is harmonized with the classical higher-order Fr\'echet…
At present, deep learning based methods are being employed to resolve the computational challenges of high-dimensional partial differential equations (PDEs). But the computation of the high order derivatives of neural networks is costly,…
State-of-the-art methods in convex and non-convex optimization employ higher-order derivative information, either implicitly or explicitly. We explore the limitations of higher-order optimization and prove that even for convex optimization,…
Inverse iteration is known to be an effective method for computing eigenvectors corresponding to simple and well-separated eigenvalues. In the non-symmetric case, the solution of shifted Hessenberg systems is a central step. Existing…
Second-order methods are emerging as promising alternatives to standard first-order optimizers such as gradient descent and ADAM for training neural networks. Though the advantages of including curvature information in computing…
We aim at computing the derivative of the solution to a parametric optimization problem with respect to the involved parameters. For a class broader than that of strongly convex functions, this can be achieved by automatic differentiation…