Related papers: Getting in the Zone for Successful Scalability
In machine learning, the scaling law describes how the model performance improves with the model and data size scaling up. From a learning theory perspective, this class of results establishes upper and lower generalization bounds for a…
Growth patterns of complex systems predict how they change in sizes, numbers, masses, etc. Understanding growth is important, especially for many biological, ecological, urban, and socioeconomic systems. One noteworthy growth behavior is…
The objective of statistical physics is to understand macroscopic behavior of a many-body system from the interactions of the constituents of that system. When many-body systems reach critical states, simple universal and scaling behaviors…
Understanding urban mobility requires models that capture how people interact with and navigate the built environment. We present a scalable, generalizable agent-based framework in which daily schedules emerge from the interplay between…
Enforcing universal symmetries in machine learning (ML) models is a common strategy to mitigate data scarcity. We show that exploiting exact, as well as approximate, label symmetries can benefit scaling laws. We illustrate the idea for the…
Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can…
Scaling laws have emerged as a unifying lens for understanding and guiding the training of large language models (LLMs). However, existing studies predominantly focus on the final-step loss, leaving open whether the entire loss dynamics…
We demonstrate that conventional artificial deep neural networks operating near the phase boundary of the signal propagation dynamics, also known as the edge of chaos, exhibit universal scaling laws of absorbing phase transitions in…
Neural scaling laws, which in some domains can predict the performance of large neural networks as a function of model, data, and compute scale, are the cornerstone of building foundation models in Natural Language Processing and Computer…
This paper proposes a specific type of Local Linear Model, the Shuffled Linear Model (SLM), that can be used as a universal approximator. Local operating points are chosen randomly and linear models are used to approximate a function or…
We apply scaling and the theory of the fundamental limits of the second-order molecular susceptibility to identify material classes with ultralarge nonlinear-optical response. Size effects are removed by normalizing all nonlinearities to…
One of the most impressive features of continuous phase transitions is the concept of universality, that allows to group the great variety of different critical phenomena into a small number of universality classes. All systems belonging to…
The System Usability Scale (SUS) is a short, survey-based approach used to determine the usability of a system from an end user perspective once a prototype is available for assessment. Individual scores are gathered using a 10-question…
We consider two different systems exhibiting a continuous phase transition into an absorbing state. Both models belong to the same universality class, i.e., they are characterized by the same scaling functions and the same critical…
What scaling limits govern neural network training dynamics when model size and training time grow in tandem? We show that despite the complex interactions between architecture, training algorithms, and data, compute-optimally trained…
Generalizable agents should adapt to diverse tasks and unseen environments beyond their training distribution. This position paper argues that such generalization requires environment scaling: expanding the distribution of executable…
Neural scaling laws have revolutionized the design and optimization of large-scale AI models by revealing predictable relationships between model size, dataset volume, and computational resources. Early research established power-law…
In high performance computing environments, we observe an ongoing increase in the available numbers of cores. This development calls for re-emphasizing performance (scalability) analysis and speedup laws as suggested in the literature…
With the spread of multi- and many-core processors more and more typical task is to re-implement some source code written originally for a single processor to run on more than one cores. Since it is a serious investment, it is important to…
Scaling laws arise and are eulogized across disciplines from natural to social sciences for providing pithy, quantitative, `scale-free', and `universal' power law relationships between two variables. On a log-log plot, the power laws…