Related papers: Gradient and Hessian approximations in Derivative …
We derived the formulae of central differentiation for the finding of the first and second derivatives of functions given in discrete points, with the number of points being arbitrary. The obtained formulae for the derivative calculation do…
Differentiable optimization layers enable learning systems to make decisions by solving embedded optimization problems. However, computing gradients via implicit differentiation requires solving a linear system with Hessian terms, which is…
Second-order optimization uses curvature information about the objective function, which can help in faster convergence. However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a…
In this work we derive a second-order approach to bilevel optimization, a type of mathematical programming in which the solution to a parameterized optimization problem (the "lower" problem) is itself to be optimized (in the "upper"…
Bilevel optimization has arisen as a powerful tool in modern machine learning. However, due to the nested structure of bilevel optimization, even gradient-based methods require second-order derivative approximations via Jacobian- or/and…
An algorithm is proposed for solving optimization problems arising in neural network training for supervised learning. The unique feature of the algorithm is the use of an auxiliary loss, in addition to the original loss employed for model…
We here adapt an extended version of the adaptive cubic regularisation method with dynamic inexact Hessian information for nonconvex optimisation in [3] to the stochastic optimisation setting. While exact function evaluations are still…
In this work, multi-variable derivative-free optimization algorithms for unconstrained optimization problems are developed. A novel procedure for approximating the gradient of multi-variable objective functions based on non-commutative maps…
Extrapolation is a well-known technique for solving convex optimization and variational inequalities and recently attracts some attention for non-convex optimization. Several recent works have empirically shown its success in some machine…
Solving initial value problems and boundary value problems of Linear Ordinary Differential Equations (ODEs) plays an important role in many applications. There are various numerical methods and solvers to obtain approximate solutions…
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are…
We introduce a generalized finite difference method for solving a large range of fully nonlinear elliptic partial differential equations in three dimensions. Methods are based on Cartesian grids, augmented by additional points carefully…
The optimal value function is one of the basic objects in the field of mathematical optimization, as it allows the evaluation of the variations in the cost/revenue generated while minimizing/maximizing a given function under some…
Large scale optimization problems are ubiquitous in machine learning and data analysis and there is a plethora of algorithms for solving such problems. Many of these algorithms employ sub-sampling, as a way to either speed up the…
Differentiable programming is revolutionizing computational science by enabling automatic differentiation (AD) of numerical simulations. While first-order gradients are well-established, second-order derivatives (Hessians) for implicit…
Quadratic interpolation models and simplex derivatives are fundamental tools in numerical optimization, particularly in derivative-free optimization. When constructed in suitably chosen affine subspaces, these tools have been shown to be…
We develop an algorithm that combines model-based and model-free methods for solving a nonlinear optimal control problem with a quadratic cost in which the system model is given by a linear state-space model with a small additive nonlinear…
Gaussian processes (GPs) with derivatives are useful in many applications, including Bayesian optimization, implicit surface reconstruction, and terrain reconstruction. Fitting a GP to function values and derivatives at $n$ points in $d$…
The use of neural networks to approximate partial differential equations (PDEs) has gained significant attention in recent years. However, the approximation of PDEs with localised phenomena, e.g., sharp gradients and singularities, remains…
Inverse problem or parameter estimation of ordinary differential equations (ODEs), the iterative process of minimizing the mismatch between model-predicted and experimental states by tuning the parameter values within an optimization…