Related papers: How Long Does Infinite Width Last? Signal Propagat…
We study the learning ability of linear recurrent neural networks with Gradient Descent. We prove the first theoretical guarantee on linear RNNs to learn any stable linear dynamic system using any a large type of loss functions. For an…
Substantial work indicates that the dynamics of neural networks (NNs) is closely related to their initialization of parameters. Inspired by the phase diagram for two-layer ReLU NNs with infinite width (Luo et al., 2021), we make a step…
We show that resonant frequencies of a system of coupled resonators in a truncated periodic lattice converge to the essential spectrum of corresponding infinite lattice. We use the capacitance matrix as a model for fully coupled resonators…
This work studies the throughput scaling laws of ad hoc wireless networks in the limit of a large number of nodes. A random connections model is assumed in which the channel connections between the nodes are drawn independently from a…
In practice, multi-task learning (through learning features shared among tasks) is an essential property of deep neural networks (NNs). While infinite-width limits of NNs can provide good intuition for their generalization behavior, the…
We analyze long range wave propagation in three-dimensional random waveguides. The waves are trapped by top and bottom boundaries, but the medium is unbounded in the two remaining directions. We consider scalar waves, and motivated by…
Recent work by Jacot et al. (2018) has shown that training a neural network using gradient descent in parameter space is related to kernel gradient descent in function space with respect to the Neural Tangent Kernel (NTK). Lee et al. (2019)…
Obtaining theoretical guarantees for neural networks training appears to be a hard problem in a general case. Recent research has been focused on studying this problem in the limit of infinite width and two different theories have been…
This paper presents a comprehensive analysis of the degree statistics in models for growing networks where new nodes enter one at a time and attach to one earlier node according to a stochastic rule. The models with uniform attachment,…
Language models with recurrent depth, also referred to as universal or looped when considering transformers, are defined by the capacity to increase their computation through the repetition of layers. Recent efforts in pretraining have…
Deep ResNets are recognized for achieving state-of-the-art results in complex machine learning tasks. However, the remarkable performance of these architectures relies on a training procedure that needs to be carefully crafted to avoid…
Dynamical processes taking place on networks have received much attention in recent years, especially on various models of random graphs (including small world and scale free networks). They model a variety of phenomena, including the…
In this paper, we initiate the study of wave propagation in a recently proposed mathematical model for stretch-limited elastic strings. We consider the longitudinal motion of a simple class of uniform, semi-infinite, stretch-limited strings…
How neural network behaves during the training over different choices of hyperparameters is an important question in the study of neural networks. In this work, inspired by the phase diagram in statistical mechanics, we draw the phase…
Despite strong performance on a variety of tasks, neural sequence models trained with maximum likelihood have been shown to exhibit issues such as length bias and degenerate repetition. We study the related issue of receiving…
Models of how things spread often assume that transmission mechanisms are fixed over time. However, social contagions--the spread of ideas, beliefs, innovations--can lose or gain in momentum as they spread: ideas can get reinforced, beliefs…
It has been known for a long time that initializing weight matrices to be orthogonal instead of having i.i.d. Gaussian components can improve training performance. This phenomenon can be analyzed using finite-width corrections, where the…
Capacity scaling laws are analyzed in an underwater acoustic network with $n$ regularly located nodes on a square. A narrow-band model is assumed where the carrier frequency is allowed to scale as a function of $n$. In the network, we…
Infinitesimal and finite amplitude error propagation in spatially extended systems are numerically and theoretically investigated. The information transport in these systems can be characterized in terms of the propagation velocity of…
How big is the risk that a few initial failures of nodes in a network amplify to large cascades that span a substantial share of all nodes? Predicting the final cascade size is critical to ensure the functioning of a system as a whole. Yet,…