Related papers: Scaling laws for amplitude surrogates
Scaling laws provide important insights that can guide the design of large language models (LLMs). Existing work has primarily focused on studying scaling laws for pretraining (upstream) loss. However, in transfer learning settings, in…
Nowadays there is an increasing interest of physicists in finding regularities related to social phenomena. This interest is clearly motivated by applications that a statistical mechanical description of the human behavior may have in our…
The method of constrained randomisation is applied to three-dimensional simulated galaxy distributions. With this technique we generate for a given data set surrogate data sets which have the same linear properties as the original data…
Training compute is increasingly outpacing the availability of high-quality data. This shifts the central challenge from optimal compute allocation to extracting maximum value from limited data. The widely adopted Chinchilla scaling law…
Empirically-determined scaling laws have been broadly successful in predicting the evolution of large machine learning models with training data and number of parameters. As a consequence, they have been useful for optimizing the allocation…
We discuss the universal scaling laws of order parameter fluctuations in any system in which the second-order critical behavior can be identified. These scaling laws can be derived rigorously for equilibrium systems when combined with the…
Scaling Laws have emerged as a powerful framework for understanding how model performance evolves as they increase in size, providing valuable insights for optimizing computational resources. In the realm of Sequential Recommendation (SR),…
Autonomous neural systems must efficiently process information in a wide range of novel environments, which may have very different statistical properties. We consider the problem of how to optimally distribute receptors along a…
Scaling laws are well studied for language models and first-stage retrieval, but not for reranking. We present the first systematic study of scaling laws for cross-encoder rerankers across pointwise, pairwise, and listwise objectives.…
Scaling laws play a central role in the success of Large Language Models (LLMs), enabling the prediction of model performance relative to compute budgets prior to training. While Transformers have been the dominant architecture, recent…
We show how strain stiffening affects the elastic response to internal forces, caused either by material defects and inhomogeneities or by active forces that molecular motors generate in living cells. For a spherical force dipole in a…
Recent studies have found that model performance has a smooth power-law relationship, or scaling laws, with training data and model size, for a wide range of problems. These scaling laws allow one to choose nearly optimal data and model…
Statistical learning algorithms are finding more and more applications in science and technology. Atomic-scale modeling is no exception, with machine learning becoming commonplace as a tool to predict energy, forces and properties of…
Modular neural networks outperform nonmodular neural networks on tasks ranging from visual question answering to robotics. These performance improvements are thought to be due to modular networks' superior ability to model the compositional…
With the analysis of the hierarchical scheme, the potential influence of the pre-constant in deriving scaling laws is exposed. It is found that a modified hierarchical scheme can achieve a throughput arbitrarily times higher than the…
Recently, 1-bit Large Language Models (LLMs) have emerged, showcasing an impressive combination of efficiency and performance that rivals traditional LLMs. Research by Wang et al. (2023); Ma et al. (2024) indicates that the performance of…
Hyperparameter optimization is an important subfield of machine learning that focuses on tuning the hyperparameters of a chosen algorithm to achieve peak performance. Recently, there has been a stream of methods that tackle the issue of…
Our goal is to analyze performance of stable linear dynamical networks subject to external stochastic disturbances. The square of the $\mathcal H_2$-norm of the network is used as a performance measure to quantify the expected steady-state…
The era of huge data necessitates highly efficient machine learning algorithms. Many common machine learning algorithms, however, rely on computationally intensive subroutines that are prohibitively expensive on large datasets. Oftentimes,…
Imposing known physical constraints, such as conservation laws, during neural network training introduces an inductive bias that can improve accuracy, reliability, convergence, and data efficiency for modeling physical dynamics. While such…