Related papers: Invariance Properties of the Natural Gradient in O…
We consider the dynamics of gradient descent (GD) in overparameterized single hidden layer neural networks with a squared loss function. Recently, it has been shown that, under some conditions, the parameter values obtained using GD achieve…
Multivariate spatial field data are increasingly common and whose modeling typically relies on building cross-covariance functions to describe cross-process relationships. An alternative viewpoint is to model the matrix of spectral…
In order to develop a differential calculus for error propagation we study local Dirichlet forms on probability spaces with square field operator $\Gamma$ -- i.e. error structures -- and we are looking for an object related to $\Gamma$…
Adaptive gradient methods have achieved remarkable success in training deep neural networks on a wide variety of tasks. However, not much is known about the mathematical and statistical properties of this family of methods. This work aims…
Classical optimisation theory guarantees monotonic objective decrease for gradient descent (GD) when employed in a small step size, or ``stable", regime. In contrast, gradient descent on neural networks is frequently performed in a large…
This paper explores variants of the subspace iteration algorithm for computing approximate invariant subspaces. The standard subspace iteration approach is revisited and new variants that exploit gradient-type techniques combined with a…
We study the implicit bias of generic optimization methods, such as mirror descent, natural gradient descent, and steepest descent with respect to different potentials and norms, when optimizing underdetermined linear regression or…
It is important in many applications to be able to extend the (outer) unit normal vector field from a hypersurface to its neighborhood in such a way that the result is a unit gradient field. The aim of the paper is to provide an elementary…
We propose a new technique that boosts the convergence of training generative adversarial networks. Generally, the rate of training deep models reduces severely after multiple iterations. A key reason for this phenomenon is that a deep…
In many relevant cases -- e.g., in hamiltonian dynamics -- a given vector field can be characterized by means of a variational principle based on a one-form. We discuss how a vector field on a manifold can also be characterized in a similar…
As the demand to integrate Artificial Intelligence into high-stakes environments continues to grow, explaining the reasoning behind neural-network predictions has shifted from a theoretical curiosity to a strict operational requirement. Our…
While the optimization landscape of policy gradient methods has been recently investigated for partially observed linear systems in terms of both static output feedback and dynamical controllers, they only provide convergence guarantees to…
The paper proposes a variational-inequality based primal-dual dynamic that has a globally exponentially stable saddle-point solution when applied to solve linear inequality constrained optimization problems. A Riemannian geometric framework…
We study the convergence of several natural policy gradient (NPG) methods in infinite-horizon discounted Markov decision processes with regular policy parametrizations. For a variety of NPGs and reward functions we show that the…
We consider the effect of a random longitudinal field on the Ising model in a transverse magnetic field. For spatial dimension $d > 2$, there is at low strength of randomness and transverse field, a phase with true long range order which is…
This paper presents enhancement strategies for the Hermitian and skew-Hermitian splitting method based on gradient iterations. The spectral properties are exploited for the parameter estimation, often resulting in a better convergence. In…
We consider a general formulation of gradient flow evolution for problems whose natural framework is the one of metric spaces. The applications we deal with are concerned with the evolution of {\it capacitary measures} with respect to the…
What is the optimal way to approximate a high-dimensional diffusion process by one in which the coordinates are independent? This paper presents a construction, called the \emph{independent projection}, which is optimal for two natural…
This paper introduces a method for efficiently approximating the inverse of the Fisher information matrix, a crucial step in achieving effective variational Bayes inference. A notable aspect of our approach is the avoidance of analytically…
We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that…