Related papers: Finite size scaling in neural networks
We consider gradient-based optimisation of wide, shallow neural networks, where the output of each hidden node is scaled by a positive parameter. The scaling parameters are non-identical, differing from the classical Neural Tangent Kernel…
We introduce a new combinatorial structure: the superselector. We show that superselectors subsume several important combinatorial structures used in the past few years to solve problems in group testing, compressed sensing, multi-channel…
Using single cluster flip Monte Carlo simulations we accurately determine new finite size scaling functions which are expressed only in terms the variable $x = \xi_L / L$, where $\xi_L$ is the correlation length in a finite system of size…
We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks…
Gardner's analysis of the optimal storage capacity of neural networks is extended to study finite-temperature effects. The typical volume of the space of interactions is calculated for strongly-diluted networks as a function of the storage…
Learning with an artificial neural network encodes the system behavior in a feed-forward function with a number of parameters optimized by data-driven training. An open question is whether one can minimize the network complexity without…
Neural networks have proven to be extremely powerful tools for modern artificial intelligence applications, but computational and storage complexity remain limiting factors. This paper presents two compatible contributions towards reducing…
We study a simple extended model of oscillator neural networks capable of storing sparsely coded phase patterns, in which information is encoded both in the mean firing rate and in the timing of spikes. Applying the methods of statistical…
We study the scaling properties of Higgs-Yukawa models. Using the technique of Finite-Size Scaling, we are able to derive scaling functions that describe the observables of the model in the vicinity of a Gaussian fixed point. A feasibility…
Cuckoo hashing with a stash is a robust multiple choice hashing scheme with high memory utilization that can be used in many network device applications. Unfortunately, for memory loads beyond 0.5, little is known on its performance. In…
We derive the calculation of two critical numbers predicting the behavior of perceptron networks. First, we derive the calculation of what we call the lossless memory (LM) dimension. The LM dimension is a generalization of the…
Finite-size scaling expressions for the current near the continuous phase transition, and for the local density near the first-order transition, are found in the steady state of the one-dimensional fully asymmetric simple-exclusion process…
We give an intuitive geometric explanation for the apparent breakdown of standard finite-size scaling in systems with periodic boundaries above the upper critical dimension. The Ising model and self-avoiding walk are simulated on…
We propose a ``half'' layer of hidden units that has some of its weights randomly set and some of them trained. A half unit is composed of two stages: First, it takes a weighted sum of its inputs with fixed random weights, and second, the…
It has been shown in recent years that the stochastic block model (SBM) is sometimes undetectable in the sparse limit, i.e., that no algorithm can identify a partition correlated with the partition used to generate an instance, if the…
The study of the distribution of volumes associated to the internal representations of learning examples allows us to derive the critical learning capacity ($\alpha_c=\frac{16}{\pi} \sqrt{\ln K}$) of large committee machines, to verify the…
We propose a novel finite size scaling analysis for percolation transition observed in complex networks. While it is known that cooperative systems in growing networks often undergo an infinite order transition with inverted…
According to the concept of typicality, an ensemble average can be accurately approximated by an expectation value with respect to a single pure state drawn at random from a high-dimensional Hilbert space. This random-vector approximation,…
The ground state and low T behavior of two-dimensional spin systems with discrete binary couplings are subtle but can be analyzed using exact computations of finite volume partition functions. We first apply this approach to Villain's fully…
Empirically-determined scaling laws have been broadly successful in predicting the evolution of large machine learning models with training data and number of parameters. As a consequence, they have been useful for optimizing the allocation…