Related papers: A matrix algebra approach to approximate Hessians
Bilevel optimization has arisen as a powerful tool in modern machine learning. However, due to the nested structure of bilevel optimization, even gradient-based methods require second-order derivative approximations via Jacobian- or/and…
Quasi-Newton methods form an important class of methods for solving nonlinear optimization problems. In such methods, first order information is used to approximate the second derivative. The aim is to mimic the fast convergence that can be…
We consider a scalar function depending on a numerical solution of an initial value problem, and its second-derivative (Hessian) matrix for the initial value. The need to extract the information of the Hessian or to solve a linear system…
Fast approximations to matrix multiplication have the potential to dramatically reduce the cost of neural network inference. Recent work on approximate matrix multiplication proposed to replace costly multiplications with table-lookups by…
The main focus in this paper is exact linesearch methods for minimizing a quadratic function whose Hessian is positive definite. We give a class of limited-memory quasi-Newton Hessian approximations which generate search directions parallel…
Computation of the trace of a matrix function plays an important role in many scientific computing applications, including applications in machine learning, computational physics (e.g., lattice quantum chromodynamics), network analysis and…
Many problems of theoretical and practical interest involve finding a convex or concave function. For instance, optimization problems such as finding the projection on the convex functions in $H^k(\Omega)$, or some problems in economics. In…
In this work we develop Curvature Propagation (CP), a general technique for efficiently computing unbiased approximations of the Hessian of any function that is computed using a computational graph. At the cost of roughly two gradient…
Machine learning (ML) problems are often posed as highly nonlinear and nonconvex unconstrained optimization problems. Methods for solving ML problems based on stochastic gradient descent are easily scaled for very large problems but may…
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are…
The dominant cost in solving least-square problems using Newton's method is often that of factorizing the Hessian matrix over multiple values of the regularization parameter ($\lambda$). We propose an efficient way to interpolate the…
In general, matrix or tensor-valued functions are approximated using the method developed for vector-valued functions by transforming the matrix-valued function into vector form. This paper proposes a tensor-based interpolation method to…
For many inverse parameter problems for partial differential equations in which the domain contains only well-separated objects, an asymptotic solution to the forward problem involving 'polarization tensors' exists. These are functions of…
A new algorithm to approximate Hermitian matrices by positive semidefinite Hermitian matrices based on modified Cholesky decompositions is presented. In contrast to existing algorithms, this algorithm allows to specify bounds on the…
Matrix functions are a central topic of linear algebra, and problems requiring their numerical approximation appear increasingly often in scientific computing. We review various limited-memory methods for the approximation of the action of…
In this paper, several modifications are introduced to the functional approximation method iterLap to reduce the approximation error, including stopping rule adjustment, proposal of new residual function, starting point selection for…
An algorithm is proposed for solving optimization problems arising in neural network training for supervised learning. The unique feature of the algorithm is the use of an auxiliary loss, in addition to the original loss employed for model…
A selection of algorithms for the rational approximation of matrix-valued functions are discussed, including variants of the interpolatory AAA method, the RKFIT method based on approximate least squares fitting, vector fitting, and a method…
Singular and oscillatory functions feature in numerous applications. The high-accuracy approximation of such functions shall greatly help us develop high-order methods for solving applied mathematics problems. This paper demonstrates that…
We present a kernel-independent method that applies hierarchical matrices to the problem of maximum likelihood estimation for Gaussian processes. The proposed approximation provides natural and scalable stochastic estimators for its…