Related papers: Scaling laws for amplitude surrogates
Geometric constraints impact the formation of a broad range of spatial networks, from amino acid chains folding to proteins structures to rearranging particle aggregates. How the network of interactions dynamically self-organizes in such…
Neural scaling laws predict how language model performance improves with increased training inputs. While aggregate metrics like validation loss can follow smooth power-law curves, individual downstream tasks exhibit diverse scaling…
For a given distribution, learning algorithm, and performance metric, the rate of convergence (or data-scaling law) is the asymptotic behavior of the algorithm's test performance as a function of number of train samples. Many learning…
The scaling law is a notable property of neural network models and has significantly propelled the development of large language models. Scaling laws hold great promise in guiding model design and resource allocation. Recent research…
Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. However, cross-entropy provides only a partial view of…
As language models scale, the amount of data they require grows -- yet many target data sources, such as low-resource languages or specialized domains, are inherently limited in size. A common strategy is to mix this scarce but valuable…
Recent theories suggest that Neural Scaling Laws arise whenever the task is linearly decomposed into power-law distributed units. Alternatively, scaling laws also emerge when data exhibit a hierarchically compositional structure, as is…
We explore the impact of parameter sparsity on the scaling behavior of Transformers trained on massive datasets (i.e., "foundation models"), in both vision and language domains. In this setting, we identify the first scaling law describing…
Generalization abilities of well-trained large language models (LLMs) are known to scale predictably as a function of model size. In contrast to the existence of practical scaling laws governing pre-training, the quality of LLMs after…
Increasing the size of a Transformer does not always lead to enhanced performance. This phenomenon cannot be explained by the empirical scaling laws. Furthermore, the model's enhanced performance is closely associated with its memorization…
We apply scaling and the theory of the fundamental limits of the second-order molecular susceptibility to identify material classes with ultralarge nonlinear-optical response. Size effects are removed by normalizing all nonlinearities to…
Random fields are commonly used for modeling of spatially (or timely) dependent stochastic processes. In this study, we provide a characterization of the intrinsic complexity of a random field in terms of its second order statistics, e.g.,…
Performance of standard processes over large distributed networks typically scales with the size of the network. For example, in planar topologies where nodes communicate with their natural neighbors, the scaling factor is $O(n)$, where $n$…
We study the emergence of a power law distribution in the systems which can be characterized by a hierarchically organized supplying network. It is shown that conservation laws on the branches of the network can, at some approximation,…
This paper derives "scaling laws"--empirical relationships between the training compute of Large Language Models (LLMs) and their performance--for economic outcomes. In a preregistered online experiment, 300 professional translators…
In scalable machine learning systems, model training is often parallelized over multiple nodes that run without tight synchronization. Most analysis results for the related asynchronous algorithms use an upper bound on the information…
It has recently been shown that the expected energy requirements of a control action applied to a complex network scales exponentially with the number of nodes that are targeted. While the exponential scaling law provides an adequate…
We propose a simple mechanism by which scaling laws emerge from feature learning in multi-layer networks. We study a high-dimensional hierarchical target that is a globally high-degree function, but that can be represented by a combination…
Critical states are sometimes identified experimentally through power-law statistics or universal scaling functions. We show here that such features naturally emerge from networks in self-sustained irregular regimes away from criticality.…
Neural scaling laws have garnered significant interest due to their ability to predict model performance as a function of increasing parameters, data, and compute. In this work, we propose a simple statistical ansatz based on memorization…