Related papers: Understanding scaling through history-dependent pr…
We consider the stochastic patterns of a system of communicating, or coupled, self-propelled particles in the presence of noise and communication time delay. For sufficiently large environmental noise, there exists a transition between a…
Large language models (LLMs) have achieved impressive results on multi-step mathematical reasoning, yet at the cost of high computational overhead. This challenge is particularly acute for test-time scaling methods such as parallel…
This paper studies the limits of language models' statistical learning in the context of Zipf's law. First, we demonstrate that Zipf-law token distribution emerges irrespective of the chosen tokenization. Second, we show that Zipf…
A generic communication model of a boolean network with transmission errors is proposed to explore the power-law scaling of states' evolution in small-world networks. In the model, the power spectrum of the population difference between…
Statistical system models provide the basis for the examination of various sorts of distributions. Classification distributions are a very common and versatile form of statistics in e.g. real economic, social, and IT systems. The…
A curious observation was made that the rank statistics of scientific citation numbers follows Zipf-Mandelbrot's law. The same pow-like behavior is exhibited by some simple random citation models. The observed regularity indicates not so…
Power-law distributions are typical macroscopic features occurring in almost all complex systems observable in nature. As a result, researchers in quantitative analyses must often generate random synthetic variates obeying power-law…
Scale independence is a ubiquitous feature of complex systems which implies a highly skewed distribution of resources with no characteristic scale. Research has long focused on why systems as varied as protein networks, evolution and stock…
When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L \propto N^{-\alpha}$ in the number of network parameters $N$. This empirical scaling law holds for a wide variety of data modalities, and may…
Noise and spatial degrees of freedom characterize most ecosystems. Some aspects of their influence on the coevolution of populations with cyclic interspecies competition have been demonstrated in recent experiments [e.g. B. Kerr et al.,…
The frequencies at which individual words occur across languages follow power law distributions, a pattern of findings known as Zipf's law. A vast literature argues over whether this serves to optimize the efficiency of human communication,…
Natural language exhibits statistical dependencies at a wide range of scales. For instance, the mutual information between words in natural language decays like a power law with the temporal lag between them. However, many statistical…
From benign overfitting in overparameterized models to rich power-law scalings in performance, simple ridge regression displays surprising behaviors sometimes thought to be limited to deep neural networks. This balance of phenomenological…
Sampling-based search, a simple paradigm for utilizing test-time compute, involves generating multiple candidate responses and selecting the best one -- typically by having models self-verify each response for correctness. In this paper, we…
One of the ultimate goals for linguists is to find universal properties in human languages. Although words are generally considered as representing arbitrary mapping between linguistic forms and meanings, we propose a new universal law that…
Spike sorting is a class of algorithms used in neuroscience to attribute the time occurences of particular electric signals, called action potential or spike, to neurons. We rephrase this problem as a particular optimization problem : Lasso…
Using statistical physics methods, we study generative diffusion models in the regime where the dimension of space and the number of data are large, and the score function has been trained optimally. Our analysis reveals three distinct…
In this paper we try to model certain features of human language complexity by means of advanced concepts borrowed from statistical mechanics. We use a time series approach, the diffusion entropy method (DE), to compute the complexity of an…
There is a recent trend in machine learning to increase model quality by growing models to sizes previously thought to be unreasonable. Recent work has shown that autoregressive generative models with cross-entropy objective functions…
The phenomenon of Stochastic Resonance (SR) is reported in a completely noise-free situation, with the role of thermal noise being taken by low-dimensional chaos. A one-dimensional, piecewise linear map and a pair of coupled…