Related papers: Scaling Laws in Jet Classification

Scaling laws provide important insights that can guide the design of large language models (LLMs). Existing work has primarily focused on studying scaling laws for pretraining (upstream) loss. However, in transfer learning settings, in…

The rank-size regularity known as Zipf's law is one of scaling laws and frequently observed within the natural living world and in social institutions. Many scientists tried to derive the rank-size scaling relation by entropy-maximizing…

Code large language models (Code LLMs) are powerful but costly to train, with scaling laws predicting performance from model size, data, and compute. However, different programming languages (PLs) have varying impacts during pre-training…

Scaling laws are a critical component of the LLM development pipeline, most famously as a way to forecast training decisions such as 'compute-optimally' trading-off parameter count and dataset size, alongside a more recent growing list of…

Transformers have become the primary architecture for natural language processing. In this study, we explore their use for auto-regressive density estimation in high-energy jet physics, which involves working with a high-dimensional space.…

Within the scaling laws paradigm, which underpins the training of large neural networks like ChatGPT and Llama, we consider a supervised regression setting and establish the existance of a strong form of the model collapse phenomenon, a…

Despite recent advancements of large language models (LLMs), optimally predicting the model size for LLM pretraining or allocating optimal resources still remains a challenge. Several efforts have addressed the challenge by proposing…

Disclosure of scaling relationship between observable quantities gives direct information about dynamics of natural phenomenon. This is the main reason why scaling plays a key role in the methodology of natural sciences. In this talk, Part…

Large genomic and imaging datasets can be used to train models that learn meaningful representations of cellular systems. Across domains, model performance improves predictably with dataset size and compute budget, providing a basis for…

We develop task scaling laws and model ladders to predict the individual task performance of pretrained language models (LMs) in the overtrained setting. Standard power laws for language modeling loss cannot accurately model task…

Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate $\eta$ and weight decay $\lambda$. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and…

At the extreme energies of the Large Hadron Collider, massive particles can be produced at such high velocities that their hadronic decays are collimated and the resulting jets overlap. Deducing whether the substructure of an observed jet…

The correlation length $\xi$, a key quantity in glassy dynamics, can now be precisely measured for spin glasses both in experiments and in simulations. However, known analysis methods lead to discrepancies either for large external fields…

Recently, 1-bit Large Language Models (LLMs) have emerged, showcasing an impressive combination of efficiency and performance that rivals traditional LLMs. Research by Wang et al. (2023); Ma et al. (2024) indicates that the performance of…

We study the growth dynamics of the size of manufacturing firms considering competition and normal distribution of competency. We start with the fact that all components of the system struggle with each other for growth as happened in real…

Neural network-based emulators for the inference of stellar parameters and elemental abundances represent an increasingly popular methodology in modern spectroscopic surveys. However, these approaches are often constrained by their…

We investigate how different compression techniques -- such as weight and activation quantization, and weight sparsity -- affect the scaling behavior of large language models (LLMs) during pretraining. Building on previous work showing that…

We present new scaling laws for the thrust production and power consumption of three-dimensional combined heaving and pitching hydrofoils by extending the three-dimensional pitching scaling laws introduced by Ayancik et al. (2019). New…

Data-driven models are revolutionizing weather forecasting. To optimize training efficiency and model performance, this paper analyzes empirical scaling laws within this domain. We investigate the relationship between model performance…

The remarkable success of large language pretraining and the discovery of scaling laws signify a paradigm shift in machine learning. Notably, the primary objective has evolved from minimizing generalization error to reducing approximation…