Related papers: Scaling laws for amplitude surrogates

Allometric scaling can reflect underlying mechanisms, dynamics and structures in complex systems; examples include typical scaling laws in biology, ecology and urban development. In this work, we study allometric scaling in scientific…

Neural networks for LHC physics have to be accurate, reliable, and controlled. Using neural surrogates for the prediction of loop amplitudes as a use case, we first show how activation functions are systematically tested with…

We found that models of evolving random networks exhibit dynamic scaling similar to scaling of growing surfaces. It is demonstrated by numerical simulations of two variants of the model in which nodes are added as well as removed [Phys.…

Performance of optimization on quadratic problems sensitively depends on the low-lying part of the spectrum. For large (effectively infinite-dimensional) problems, this part of the spectrum can often be naturally represented or approximated…

Despite the fact that experimental neural scaling laws have substantially guided empirical progress in large-scale machine learning, no existing theory can quantitatively predict the exponents of these important laws for any modern LLM…

Compressed frictional granular matter cannot flow without dilation. Upon forced shearing to generate flow, the amount of dilation may depend on the initial preparation and a host of material variables. On the basis of both experiments and…

We consider the solvable neural scaling model with three parameters: data complexity, target complexity, and model-parameter-count. We use this neural scaling model to derive new predictions about the compute-limited, infinite-data scaling…

While scaling laws provide a reliable methodology for predicting train loss across compute scales for a single data distribution, less is known about how these predictions should change as we change the distribution. In this paper, we…

Efficient engineered systems require scalability. A scalable system has increasing performance with increasing system size. In an ideal case, the increase in performance (e.g., speedup) corresponds to the number of units that are added to…

To address the challenge of limited experimental materials data, extensive physical property databases are being developed based on high-throughput computational experiments, such as molecular dynamics simulations. Previous studies have…

We study scaling laws for singular perturbation problems associated with a class of two-dimensional martensitic phase transformations and deduce a domain dependence of the scaling law in the singular perturbation parameter. In these…

Many physical systems share the property of scale invariance. Most of them show ordinary power-law scaling, where quantities can be expressed as a leading power law times a scaling function which depends on scaling-invariant ratios of the…

Optimal power flow (OPF) is one of the fundamental tasks for power system operations. While machine learning (ML) approaches such as deep neural networks (DNNs) have been widely studied to enhance OPF solution speed and performance, their…

Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of…

As we scale to more massive machine learning models, the frequent synchronization demands inherent in data-parallel approaches create significant slowdowns, posing a critical challenge to further scaling. Recent work develops an approach…

Simulation results of Ising systems for several update rules, observables, and dimensions are analyzed. The lattice-size dependence is discussed for the autocorrelation times and for the weights of eigenvalues, giving fit results in the…

We introduce a scaling law for fine-tuning large language models (LLMs) under fixed compute budgets that explicitly accounts for data composition. Conventional approaches measure training data solely by total tokens, yet the number of…

Code large language models (Code LLMs) are powerful but costly to train, with scaling laws predicting performance from model size, data, and compute. However, different programming languages (PLs) have varying impacts during pre-training…

In this article we study scaling laws for simplified multi-well nucleation problems without gauge invariances which are motivated by models for shape-memory alloys. Seeking to explore the role of the order of lamination on the energy…

Surrogate models provide compact relations between user-defined input parameters and output quantities of interest, enabling the efficient evaluation of complex parametric systems in many-query settings. Such capabilities are essential in a…