Related papers: Asymptotic properties of one-layer artificial neur…
In this work we determine a process-level Large Deviation Principle (LDP) for a model of interacting particles indexed by a lattice $\mathbb{Z}^d$. The connections are random, sparse and unscaled, so that the system converges in the large…
Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to…
Effectively scaling up deep reinforcement learning models has proven notoriously difficult due to network pathologies during training, motivating various targeted interventions such as periodic reset and architectural advances such as layer…
An artificial neural network is presented based on the idea of connections between units that are only active for a specific range of input values and zero outside that range (and so are not evaluated outside the active range). The…
An artificial neural network architecture, parameterization networks, is proposed for simulating extrapolated dynamics beyond observed data in dynamical systems. Parameterization networks are used to ensure the long term integrity of…
The bivariate distribution of degrees of adjacent vertices (degree-degree distribution) is an important network characteristic defining the statistical dependencies between degrees of adjacent vertices. We show the asymptotic degree-degree…
For a large class of feature maps we provide a tight asymptotic characterisation of the test error associated with learning the readout layer, in the high-dimensional limit where the input dimension, hidden layer widths, and number of…
Asymptotic properties of random graph sequences, like occurrence of a giant component or full connectivity in Erd\H{o}s-R\'enyi graphs, are usually derived with very specific choices for defining parameters. The question arises to which…
In random graph models, the degree distribution of an individual node should be distinguished from the (empirical) degree distribution of the graph that records the fractions of nodes with given degree. We introduce a general framework to…
When the parameters are independently and identically distributed (initialized) neural networks exhibit undesirable properties that emerge as the number of layers increases, e.g. a vanishing dependency on the input and a concentration on…
In this paper we study the problem of learning a shallow artificial neural network that best fits a training data set. We study this problem in the over-parameterized regime where the number of observations are fewer than the number of…
Neuromorphic networks can be described in terms of coarse-grained variables, where emergent sustained behaviours spontaneously arise if stochasticity is properly taken in account. For example it has been recently found that a directed…
In this article we prove a Generalized Asypmtotic Equipartition Property for Networked Data Structures modelled as coloured random graphs. The main techniques in this article remains large deviation principles for suitably defined empirical…
Synchronous stochastic gradient descent (SGD) is the most common method used for distributed training of deep learning models. In this algorithm, each worker shares its local gradients with others and updates the parameters using the…
The representation of functions by artificial neural networks depends on a large number of parameters in a non-linear fashion. Suitable parameters of these are found by minimizing a 'loss functional', typically by stochastic gradient…
Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from…
We consider high-dimensional distribution estimation through autoregressive networks. By combining the concepts of sparsity, mixtures and parameter sharing we obtain a simple model which is fast to train and which achieves state-of-the-art…
In this paper, a sparsity-aware adaptive algorithm for distributed learning in diffusion networks is developed. The algorithm follows the set-theoretic estimation rationale. At each time instance and at each node of the network, a closed…
A probabilistic generative network model with $n$ nodes and $m$ overlapping layers is obtained as a superposition of $m$ mutually independent Bernoulli random graphs of varying size and strength. When $n$ and $m$ are large and of the same…
We consider sparse random intersection graphs with the property that the clustering coefficient does not vanish as the number of nodes tends to infinity. We find explicit asymptotic expressions for the correlation coefficient of degrees of…