Related papers: Consistent Computation of First- and Second-Order …
Efficiently approximating local curvature information of the loss function is a key tool for optimization and compression of deep neural networks. Yet, most existing methods to approximate second-order information have high computational or…
We present PYHESSIAN, a new scalable framework that enables fast computation of Hessian (i.e., second-order derivative) information for deep neural networks. PYHESSIAN enables fast computations of the top Hessian eigenvalues, the Hessian…
When training large models, such as neural networks, the full derivatives of order 2 and beyond are usually inaccessible, due to their computational cost. Therefore, among the second-order optimization methods, it is common to bypass the…
The computation of first and second-order derivatives is a staple in many computing applications, ranging from machine learning to scientific computing. We propose an algorithm to automatically differentiate algorithms written in a subset…
A second-order face-centred finite volume strategy on general meshes is proposed. The method uses a mixed formulation in which a constant approximation of the unknown is computed on the faces of the mesh. Such information is then used to…
The decentralized gradient descent (DGD) algorithm, and its sibling, diffusion, are workhorses in decentralized machine learning, distributed inference and estimation, and multi-agent coordination. We propose a novel, principled framework…
Dense granular flows exhibit both surface deformation and secondary flows due to the presence of normal stress differences. Yet, a complete mathematical modelling of these two features is still lacking. This paper focuses on a steady…
In quantum computing, the variational quantum algorithms (VQAs) are well suited for finding optimal combinations of things in specific applications ranging from chemistry all the way to finance. The training of VQAs with gradient descent…
We study differentiable strongly quasiconvex functions for providing new properties for algorithmic and monotonicity purposes. Furthemore, we provide insights into the decreasing behaviour of strongly quasiconvex functions, applying this…
Motivated by conforming finite element methods for elliptic problems of second order, we analyze the approximation of the gradient of a target function by continuous piecewise polynomial functions over a simplicial mesh. The main result is…
First-order optimization algorithms can be considered as a discretization of ordinary differential equations (ODEs) \cite{su2014differential}. In this perspective, studying the properties of the corresponding trajectories may lead to…
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are…
We develop efficient numerical integration methods for computing an integral whose integrand is a product of a smooth function and the Gaussian function with a small standard deviation. Traditional numerical integration methods applied to…
In the context of the analysis of measured data, one is often faced with the task to differentiate data numerically. Typically, this occurs when measured data are concerned or data are evaluated numerically during the evolution of partial…
Defined mathematically as critical points of surface area subject to a volume constraint, constant mean curvatures (CMC) surfaces are idealizations of interfaces occurring between two immiscible fluids. Their behavior elucidates phenomena…
We discuss in which sense general metric measure spaces possess a first order differential structure. Building on this, we then see that on spaces with Ricci curvature bounded from below a second order calculus can be developed, permitting…
Lebesgue integration of derivatives of strongly-oscillatory functions is a recurring challenge in computational science and engineering. Integration by parts is an effective remedy for huge computational costs associated with Monte Carlo…
We present a high-order surface quadrature (HOSQ) for accurately approximating regular surface integrals on closed surfaces. The initial step of our approach rests on exploiting square-squeezing--a homeomorphic bilinear square-simplex…
In this work we derive a second-order approach to bilevel optimization, a type of mathematical programming in which the solution to a parameterized optimization problem (the "lower" problem) is itself to be optimized (in the "upper"…
Any procedure applied to data, and any quantity derived from data, is required to respect the nature and symmetries of the data. This axiom applies to refinement procedures and multiresolution transforms as well as to more basic operations…