Related papers: Invariance Properties of the Natural Gradient in O…
In this note, we provide a important considerations of a familiar topic: the gradient of a vector field. The gradient of a vector field is a common quantity represented in continuum mechanics. However, even for Cartesian coordinate systems,…
We construct a kinetic model for matter-radiation interactions whose hydrodynamic gradient expansion can be computed analytically up to infinite order in derivatives, in the fully nonlinear regime, and for arbitrary flows. The frequency…
Hyperparameter tuning is one of the essential steps to guarantee the convergence of machine learning models. We argue that intuition about the optimal choice of hyperparameters for stochastic gradient descent can be obtained by studying a…
This paper focuses on training implicit models of infinite layers. Specifically, previous works employ implicit differentiation and solve the exact gradient for the backward propagation. However, is it necessary to compute such an exact but…
Generative modeling can be formulated as learning a mapping f such that its pushforward distribution matches the data distribution. The pushforward behavior can be carried out iteratively at inference time, for example in diffusion and…
We present here a new model and algorithm which performs an efficient Natural gradient descent for Multilayer Perceptrons. Natural gradient descent was originally proposed from a point of view of information geometry, and it performs the…
We study the autocovariance functions of moving average random fields over the integer lattice $\mathbb{Z}^d$ from an algebraic perspective. These autocovariances are parametrized polynomially by the moving average coefficients, hence…
We address the challenging problem of deep representation learning--the efficient adaption of a pre-trained deep network to different tasks. Specifically, we propose to explore gradient-based features. These features are gradients of the…
Random fields are useful mathematical objects in the characterization of non-deterministic complex systems. A fundamental issue in the evolution of dynamical systems is how intrinsic properties of such structures change in time. In this…
Information geometry provides a tool to systematically investigate parameter sensitivity of the state of a system. If a physical system is described by a linear combination of eigenstates of a complex (that is, non-Hermitian) Hamiltonian,…
In this paper, a novel second-order method called NG+ is proposed. By following the rule ``the shape of the gradient equals the shape of the parameter", we define a generalized fisher information matrix (GFIM) using the products of…
The non-invasive spatiotemporal control of cellular functions, organization of tissues, and even the behavior of small animals has become paramount for advanced therapies. As magnetic fields do not interact with biological matter, their…
Using backpropagation to compute gradients of objective functions for optimization has remained a mainstay of machine learning. Backpropagation, or reverse-mode differentiation, is a special case within the general family of automatic…
In deep learning, it is common to use more network parameters than training points. In such scenarioof over-parameterization, there are usually multiple networks that achieve zero training error so that thetraining algorithm induces an…
Information geometry is the application of differential geometry in statistics, where the Fisher-Rao metric serves as the Riemannian metric on the statistical manifold, providing an intrinsic property for parameter sensitivity. In this…
Orthogonal Gradient Descent (OGD) has emerged as a powerful method for continual learning. However, its Euclidean projections do not leverage the underlying information-geometric structure of the problem, which can lead to suboptimal…
This paper presents a theory for how geometric image transformations can be handled by a first layer of linear receptive fields, in terms of true covariance properties, which, in turn, enable geometric invariance properties at higher levels…
We present practical Levenberg-Marquardt variants of Gauss-Newton and natural gradient methods for solving non-convex optimization problems that arise in training deep neural networks involving enormous numbers of variables and huge data…
We present a new class of gradient-type optimization methods that extends vanilla gradient descent, mirror descent, Riemannian gradient descent, and natural gradient descent. Our approach involves constructing a surrogate for the objective…
The aim of this work is to learn models of population dynamics of physical systems that feature stochastic and mean-field effects and that depend on physics parameters. The learned models can act as surrogates of classical numerical models…