Related papers: The Hamilton-Jacobi Theory of Deep Learning
Learning in the brain is poorly understood and learning rules that respect biological constraints, yet yield deep hierarchical representations, are still unknown. Here, we propose a learning rule that takes inspiration from neuroscience and…
In deep learning, dense layer connectivity has become a key design principle in deep neural networks (DNNs), enabling efficient information flow and strong performance across a range of applications. In this work, we model densely connected…
We present stochastic homogenization results for viscous Hamilton-Jacobi equations using a new argument which is based only on the subadditive structure of maximal subsolutions (solutions of the "metric problem"). This permits us to give…
A main puzzle of deep networks revolves around the absence of overfitting despite large overparametrization and despite the large capacity demonstrated by zero training error on randomly labeled data. In this note, we show that the dynamics…
This paper is the first attempt to systematically study properties of the effective Hamiltonian $\overline{H}$ arising in the periodic homogenization of some coercive but nonconvex Hamilton-Jacobi equations. Firstly, we introduce a new and…
Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem…
We investigate the asymptotic behavior of solutions of Hamilton-Jacobi equations with large drift term in an open subset of two-dimensional Euclidean space. When the drift is given by $\varepsilon^{-1} (H_{x_2}, -H_{x_1})$ of a Hamiltonian…
Decentralized optimization with time-varying networks is an emerging paradigm in machine learning. It saves remarkable communication overhead in large-scale deep training and is more robust in wireless scenarios especially when nodes are…
Recent observations have been made that bridge splitting methods arising from optimization, to the Hopf and Lax formulas for Hamilton-Jacobi Equations with Hamiltonians $H(p)$. This has produced extremely fast algorithms in computing…
We propose a hierarchical training algorithm for standard feed-forward neural networks that adaptively extends the network architecture as soon as the optimization reaches a stationary point. By solving small (low-dimensional) optimization…
Hamiltonian neural networks (HNNs) are state-of-the-art models that regress the vector field of a dynamical system under the learning bias of Hamilton's equations. A recent observation is that embedding a bias regarding the additive…
This report concerns the inverse problem of estimating a spacially dependent coefficient of a partial differential equation from observations of the solution at the boundary. Such a problem can be formulated as an optimal control problem…
In this work we combine two distinct machine learning methodologies, sequential Monte Carlo and Bayesian experimental design, and apply them to the problem of inferring the dynamical parameters of a quantum system. We design the algorithm…
Training neural networks is a challenging non-convex optimization problem, and backpropagation or gradient descent can get stuck in spurious local optima. We propose a novel algorithm based on tensor decomposition for guaranteed training of…
We prove first-order convergence of semi-discrete monotone finite difference schemes for Hamilton--Jacobi equations on the Wasserstein space over a finite graph. A central challenge is the boundary degeneracy of the Wasserstein simplex,…
In this paper, for a variety of nonholonomic (reducible) Hamiltonian systems, we first give to various distributional Hamiltonian systems, by analyzing carefully the dynamics and structures of the nonholonomic Hamiltonian systems. Secondly,…
Analog crossbar arrays comprising programmable nonvolatile resistors are under intense investigation for acceleration of deep neural network training. However, the ubiquitous asymmetric conductance modulation of practical resistive devices…
We consider the well-posedness and numerical approximation of a Hamilton--Jacobi equation on an evolving hypersurface in $\mathbb R^3$. Definitions of viscosity sub- and supersolutions are extended in a natural way to evolving hypersurfaces…
Hebbian learning is a key principle underlying learning in biological neural networks. We relate a Hebbian spike-timing-dependent plasticity rule to noisy gradient descent with respect to a non-convex loss function on the probability…
The rarely used Hamilton-Jacobi equation has been utilized as an elegant way to find the trajectories of mechanical systems and to derive symplectic maps. Further, the exact solution in kick approximation of Hamilton's equations of motion…