Related papers: On Information-Theoretic Scaling Laws for Wireless…

Neural scaling laws have revolutionized the design and optimization of large-scale AI models by revealing predictable relationships between model size, dataset volume, and computational resources. Early research established power-law…

This paper analyzes the impact and benefits of infrastructure support in improving the throughput scaling in networks of $n$ randomly located wireless nodes. The infrastructure uses multi-antenna base stations (BSs), in which the number of…

In this work, we provide a sharp theory of scaling laws for two-layer neural networks trained on a class of hierarchical multi-index targets, in a genuinely representation-limited regime. We derive exact information-theoretic scaling laws…

On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is…

Connectivity correlations play an important role in the structure of scale-free networks. While several empirical studies exist, there is no general theoretical analysis that can explain the largely varying behavior of real networks. Here,…

We show that the error of iteratively magnitude-pruned networks empirically follows a scaling law with interpretable coefficients that depend on the architecture and task. We functionally approximate the error of the pruned networks,…

How does the shape of a network change as its size increases? Although random graph models provide some expectations for such "scaling behaviors" in the structure of networks, relatively little is known about how empirical network structure…

Critical, or scale independent, systems are so ubiquitous, that gaining theoretical insights on their nature and properties has many direct repercussions in social and natural sciences. In this report, we start from the simplest possible…

As neural networks continue to grow in size but datasets might not, it is vital to understand how much performance improvement can be expected: is it more important to scale network size or data volume? Thus, neural network scaling laws,…

Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can…

Scale transformations have played an extremely successful role in studies of cosmological large-scale structure by relating the non-linear spectrum of cosmological density fluctuations to the linear primordial power at longer wavelengths.…

All networks can be analyzed at multiple scales. A higher scale of a network is made up of macro-nodes: subgraphs that have been grouped into individual nodes. Recasting a network at higher scales can have useful effects, such as decreasing…

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L \propto N^{-\alpha}$ in the number of network parameters $N$. This empirical scaling law holds for a wide variety of data modalities, and may…

We consider diffusion processes on power-law small-world networks in different dimensions. In one dimension, we find a rich phase diagram, with different transient and recurrent phases, including a critical line with continuously varying…

Capacity scaling laws are analyzed in an underwater acoustic network with $n$ regularly located nodes on a square, in which both bandwidth and received signal power can be limited significantly. A narrow-band model is assumed where the…

It is commonly believed that scaling language models should commit a significant space or time cost, by increasing the parameters (parameter scaling) or output tokens (inference-time scaling). We introduce the third and more…

When training deep neural networks, a model's generalization error is often observed to follow a power scaling law dependent both on the model size and the data size. Perhaps the best known example of such scaling laws are for…

We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination…

Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and…

We consider one-hop communication in wireless networks with random connections. In the random connection model, the channel powers between different nodes are drawn from a common distribution in an i.i.d. manner. An scheme achieving the…