Related papers: Higher-order Reverse Automatic Differentiation wit…
Efficiently approximating local curvature information of the loss function is a key tool for optimization and compression of deep neural networks. Yet, most existing methods to approximate second-order information have high computational or…
In this paper, we consider the numerical pricing of financial derivatives using Radial Basis Function generated Finite Differences in space. Such discretization methods have the advantage of not requiring Cartesian grids. Instead, the nodes…
Hessian operators arising in inverse problems governed by partial differential equations (PDEs) play a critical role in delivering efficient, dimension-independent convergence for both Newton solution of deterministic inverse problems, as…
We introduce AdaSub, a stochastic optimization algorithm that computes a search direction based on second-order information in a low-dimensional subspace that is defined adaptively based on available current and past information. Compared…
Recent theoretical work on automatic differentiation (autodiff) has focused on characteristics such as correctness and efficiency while assuming that all derivatives are automatically generated by autodiff using program transformation, with…
High-order derivatives of nested functions of a single variable can be computed with the celebrated Fa\`a di Bruno's formula. Although generalizations of such formula to multiple variables exist, their combinatorial nature generates an…
This work introduces the nested-set Hessian approximation, a second-order approximation method that can be used in any derivative-free optimization routine that requires such information. It is built on the foundation of the generalized…
An algorithm is proposed for solving optimization problems arising in neural network training for supervised learning. The unique feature of the algorithm is the use of an auxiliary loss, in addition to the original loss employed for model…
Second-order Newton-type algorithms that leverage the exact Hessian or its approximation are central to solve nonlinear optimization problems. However, their applications in solving large-scale nonconvex problems are hindered by three…
Although it is widely known that Gaussian processes can be conditioned on observations of the gradient, this functionality is of limited use due to the prohibitive computational cost of $\mathcal{O}(N^3 D^3)$ in data points $N$ and…
Differentially private (stochastic) gradient descent is the workhorse of DP private machine learning in both the convex and non-convex settings. Without privacy constraints, second-order methods, like Newton's method, converge faster than…
We propose a second-order (Hessian or Hessian-free) based optimization method for variational inference inspired by Gaussian backpropagation, and argue that quasi-Newton optimization can be developed as well. This is accomplished by…
We analyze Newton's method with lazy Hessian updates for solving general possibly non-convex optimization problems. We propose to reuse a previously seen Hessian for several iterations while computing new gradients at each step of the…
A class of high-order numerical algorithms for Riesz derivatives are established through constructing new generating functions. Such new high-order formulas can be regarded as the modification of the classical (or shifted) Lubich's…
Fast prototyping of new SE(3) estimation objectives remains awkward in practice. Modern Lie-group frameworks -- GTSAM, manif, Sophus, SymForce, Ceres -- target first-order workloads through different code-generation and…
We consider Hadamard fractional derivatives and integrals of variable fractional order. A new type of fractional operator, which we call the Hadamard-Marchaud fractional derivative, is also considered. The objective is to represent these…
To efficiently implement many-particle quantum simulations on quantum computers we develop and present methods for inverting the Campbell-Baker-Hausdorff lemma to 3rd and 4th order in the commutator. That is, we reexpress exp{-i(H_1 + H_2 +…
In a multi-agent network, we consider the problem of minimizing an objective function that is expressed as the sum of private convex and smooth functions, and a (possibly) non-differentiable convex regularizer. We propose a novel…
We give an algorithm to compute the series expansion for the inverse of a given function. The algorithm is extremely easy to implement and gives the first $N$ terms of the series. We show several examples of its application in calculating…
Higher order derivatives of functions are structured high dimensional objects which lend themselves to many alternative representations, with the most popular being multi-index, matrix and tensor representations. The choice between them…