Related papers: A duality connecting neural network and cosmologic…
A fairly comprehensive analysis is presented for the gradient descent dynamics for training two-layer neural network models in the situation when the parameters in both layers are updated. General initialization schemes as well as general…
We study the training dynamics of shallow neural networks, in a two-timescale regime in which the stepsizes for the inner layer are much smaller than those for the outer layer. In this regime, we prove convergence of the gradient flow to a…
A dynamical neural network consists of a set of interconnected neurons that interact over time continuously. It can exhibit computational properties in the sense that the dynamical system's evolution and/or limit points in the associated…
Understanding the learning dynamics of neural networks is one of the key issues for the improvement of optimization algorithms as well as for the theoretical comprehension of why deep neural nets work so well today. In this paper, we…
We establish a duality relation between Hamiltonian systems and neural network-based learning systems. We show that the Hamilton's equations for position and momentum variables correspond to the equations governing the activation dynamics…
A key challenge in building theoretical foundations for deep learning is the complex optimization dynamics of neural networks, resulting from the high-dimensional interactions between the large number of network parameters. Such non-trivial…
Parameter-space and function-space provide two different duality frames in which to study neural networks. We demonstrate that symmetries of network densities may be determined via dual computations of network correlation functions, even…
The computational capabilities of a neural network are widely assumed to be determined by its static architecture. Here we challenge this view by establishing that a fixed neural structure can operate in fundamentally different…
Understanding the inductive bias and generalization properties of large overparametrized machine learning models requires to characterize the dynamics of the training algorithm. We study the learning dynamics of large two-layer neural…
Bases, mappings, projections and metrics, natural for Neural network training, are introduced. Graph-theoretical interpretation is offered. Non-Gaussianity naturally emerges, even in relatively simple datasets. Training statistics,…
One possible description for the current accelerated expansion of the universe is quintessence dynamics. The basic idea of quintessence consists of analyzing cosmological scenarios driven by scalar fields. In this work we present some…
Scalar field dynamics may give rise to a nonzero cosmological variation of fundamental constants. Within different scenarios based on the unification of gauge couplings, the various claimed observations and bounds may be combined in order…
A dynamical theory is studied in which a scalar field $\phi$ in Einstein- Minkowski space is coupled to the four-velocity $N_{\mu}$ of a preferred inertial observer in that space. As a consistent requirement on this coupling we study a…
Linear layers in neural networks (NNs) trained by gradient descent can be expressed as a key-value memory system which stores all training datapoints and the initial weights, and produces outputs using unnormalised dot attention over the…
We analyze recurrent neural networks with diagonal hidden-to-hidden weight matrices, trained with gradient descent in the supervised learning setting, and prove that gradient descent can achieve optimality \emph{without} massive…
In this chapter, we utilize dynamical systems to analyze several aspects of machine learning algorithms. As an expository contribution we demonstrate how to re-formulate a wide variety of challenges from deep neural networks, (stochastic)…
Training deep neural networks is a challenging non-convex optimization problem. Recent work has proven that the strong duality holds (which means zero duality gap) for regularized finite-width two-layer ReLU networks and consequently…
We study the dynamics of a quintessence model based on two interacting scalar fields. The model can account for the (recent) accelerated expansion of the Universe suggested by astronomical observations. Acceleration can be permanent or…
A detailed comparative qualitative analysis and numerical simulation of evolution of the cosmological models based on the doublet of classical and phantom scalar fields with self-action. The 2-dimensional and 3-dimensional projections of…
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is…