Related papers: Intriguing Properties of Quantization at Scale
Neural scaling laws define a predictable relationship between a model's parameter count and its performance after training in the form of a power law. However, most research to date has not explicitly investigated whether scaling laws can…
In the software development process, model transformation is increasingly assimilated. However, systems being developed with model transformation sometimes grow in size and become complex. Meanwhile, the performance of model transformation…
Qualitative modelling is a technique integrating the fields of theoretical computer science, artificial intelligence and the physical and biological sciences. The aim is to be able to model the behaviour of systems without estimating…
It is well known that in a quantum phase transition (QPT), entanglement remains short ranged [Osterloh et al., Nature 416 608-610 (2005)]. We ask if there is a quantum property entailing the whole system which diverges near this point.…
Effective field theories (EFT) parameterize the long-distance effects of short-distance dynamics whose details may or may not be known. It is known that EFT coefficients must obey certain positivity constraints if causality and unitarity…
Quantile regression is a tool for learning conditional distributions. In this paper we study quantile regression in the setting where a protected attribute is unavailable when fitting the model. This can lead to "unfair'' quantile…
How well do language models deal with quantification? In this study, we focus on 'few'-type quantifiers, as in 'few children like toys', which might pose a particular challenge for language models because the sentence components with out…
Coarse-grained descriptions can be used to account for physical processes in which information is lost or not entirely accessible. In this paper, we start by proposing a connection between effective, coarse-grained descriptions of quantum…
The dynamic scaling of curved interfaces presents features that are strikingly different from those of the planar ones. Spherical surfaces above one dimension are flat because the noise is irrelevant in such cases. Kinetic roughening is…
Standard evaluation protocols reveal a counterintuitive phenomenon: on 7.7% of benchmark problems spanning five datasets, larger language models underperform smaller ones by 28.4 percentage points despite 10-100x more parameters. Through…
On a variety of tasks, the performance of neural networks predictably improves with training time, dataset size and model size across many orders of magnitude. This phenomenon is known as a neural scaling law. Of fundamental importance is…
Recent successes of massively overparameterized models have inspired a new line of work investigating the underlying conditions that enable overparameterized models to generalize well. This paper considers a framework where the possibly…
Given a fixed budget for total model size, one must choose between training a single large model or combining the predictions of multiple smaller models. We investigate this trade-off for ensembles of random-feature ridge regression models…
Despite their strong performance on many tasks, pre-trained language models have been shown to struggle on out-of-distribution compositional generalization. Meanwhile, recent work has shown considerable improvements on many NLP tasks from…
Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment through quantization, pruning, or decoding strategy…
Publicly traded companies are fundamental units of contemporary economies and markets and are important mechanisms through which humans interact with their environments. Understanding the general properties that underlie the processes of…
Despite the growing prevalence of large language model (LLM) architectures, a crucial concern persists regarding their energy and power consumption, which still lags far behind the remarkable energy efficiency of the human brain. Recent…
In evolution theory the concept of a fitness landscape has played an important role, evolution itself being portrayed as a hill-climbing process on a rugged landscape. In this article it is shown that in general, in the presence of other…
Deep neural network pruning and quantization techniques have demonstrated it is possible to achieve high levels of compression with surprisingly little degradation to test set accuracy. However, this measure of performance conceals…
A careful study of the classical/quantum connection with the aid of coherent states offers new insights into various technical problems. This analysis includes both canonical as well as closely related affine quantization procedures. The…