Related papers: Getting in the Zone for Successful Scalability
Scaling laws have been used to describe how large language model (LLM) performance scales with model size, training data size, or amount of computational resources. Motivated by the fact that neural quantum states (NQS) has increasingly…
When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L \propto N^{-\alpha}$ in the number of network parameters $N$. This empirical scaling law holds for a wide variety of data modalities, and may…
We discuss the computational complexity of solving linear programming problems by means of an analog computer. The latter is modeled by a dynamical system which converges to the optimal vertex solution. We analyze various probability…
An application is a logical image of the world on a computer. A scalable application is an application that allows one to update that logical image at run time. To put it in operational terms: an application is scalable if a client can…
Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on…
Cross-embodiment generalization underpins the vision of building generalist embodied agents for any robot, yet its enabling factors remain poorly understood. We investigate embodiment scaling laws, the hypothesis that increasing the number…
Guided by the belief of the scaling law, large language models (LLMs) have achieved impressive performance in recent years. However, scaling law only gives a qualitative estimation of loss, which is influenced by various factors such as…
Scale invariance and the resulting power law behaviours are seen in diverse systems. In this work we consider translation, rotational and scale invariant systems defined on a lattice, such that the variables defining the state at every…
The assumption of Gaussian or Gaussian mixture data has been extensively exploited in a long series of precise performance analyses of machine learning (ML) methods, on large datasets having comparably numerous samples and features. To…
Learning arguably involves the discovery and memorization of abstract rules. The aim of this paper is to study associative memory mechanisms. Our model is based on high-dimensional matrices consisting of outer products of embeddings, which…
The paper presents a simple derivation of the Gustafson-Barsis law from the Amdahl's law. In the computer literature these two laws describing the speedup limits of parallel applications are derived separately. It is shown, that treating…
We study the universality of work statistics performed during a quench in gapless quantum systems. We show that the cumulants of work scale separately in the fast and slow quench regimes, following a power law analogous to the universal…
Scaling laws are powerful summaries of the variations of urban attributes with city size. However, the validity of their universal meaning for cities is hampered by the observation that different scaling regimes can be encountered for the…
Designing autonomous drone swarms is hampered by a vast design space spanning platform, algorithmic, and numerical-strength choices. We perform large-scale agent-based simulations in three canonical scenarios: swarm-on-swarm battle,…
Units equivariance (or units covariance) is the exact symmetry that follows from the requirement that relationships among measured quantities of physics relevance must obey self-consistent dimensional scalings. Here, we express this…
Recently a number of empirical "universal" scaling law papers have been published, most notably by OpenAI. `Scaling laws' refers to power-law decreases of training or test error w.r.t. more data, larger neural networks, and/or more compute.…
Analyses of urban scaling laws assume that observations in different cities are independent of the existence of nearby cities. Here we introduce generative models and data-analysis methods that overcome this limitation by modelling…
We present new scaling laws for the thrust production and power consumption of three-dimensional combined heaving and pitching hydrofoils by extending the three-dimensional pitching scaling laws introduced by Ayancik et al. (2019). New…
We perform extensive numerical simulations of different versions of the sandpile model. We find that previous claims about universality classes are unfounded, since the method previously employed to analyze the data suffered a systematic…
The quality of Large Language Model (LLM) pretraining depends on multiple factors, including the compute budget and the choice of optimization algorithm. Empirical scaling laws are widely used to predict loss as model size and training data…