Related papers: Syntactic Structures and Code Parameters
Representing token embeddings as probability distributions over learned manifolds allows for more flexible contextual inference, reducing representational rigidity while enhancing semantic granularity. Comparative evaluations demonstrate…
A computation of the dynamical structure factor of topologically disordered systems, where the disorder can be described in terms of euclidean random matrices, is presented. Among others, structural glasses and supercooled liquids belong to…
A fundamental challenge in the cognitive sciences is discovering the dynamics that govern behaviour. Take the example of spoken language, which is characterised by a highly variable and complex set of physical movements that map onto the…
Code super-optimization is the task of transforming any given program to a more efficient version while preserving its input-output behaviour. In some sense, it is similar to the paraphrase problem from natural language processing where the…
Code assistants are increasingly utilized in test-driven software development, yet the theoretical mechanisms behind their environment-interaction strategies remain underexplored. We provide a probabilistic framework for two dominant…
Recent advances on the Vector Space Model have significantly improved some NLP applications such as neural machine translation and natural language generation. Although word co-occurrences in context have been widely used in…
We discuss the dynamic behavior of a tagged particle close to a classical localization transition in the framework of the mode-coupling theory of the glass transition. Asymptotic results are derived for the order parameter as well as the…
We derive a microscopic theory of glassy dynamics based on the transport of voids by micro-string motions, each of which involves particles arranged in a line hopping simultaneously displacing one another. Disorder is modeled by a random…
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sense reasoning ability. We show, however, with a new diagnostic…
Kinetically constrained spin systems play an important role in understanding key properties of the dynamics of slowly relaxing materials, such as glasses. So far kinetic constraints have been introduced in idealised models aiming to capture…
We study by Molecular Dynamics simulation a dense one-component system of particles confined on a spherical substrate. We more specifically investigate the evolution of the structural and dynamical properties of the system when changing the…
Despite achieving impressive results on standard benchmarks, large foundational models still struggle against code-switching test cases. When data scarcity cannot be used as the usual justification for poor performance, the reason may lie…
An additive noise channel is considered, in which the distribution of the noise is nonparametric and unknown. The problem of learning encoders and decoders based on noise samples is considered. For uncoded communication systems, the problem…
Code-switching is a commonly observed communicative phenomenon denoting a shift from one language to another within the same speech exchange. The analysis of code-switched data often becomes an assiduous task, owing to the limited…
The intricate hierarchical structure of syntax is fundamental to the intricate and systematic nature of human language. This study investigates the premise that language models, specifically their attention distributions, can encapsulate…
The integration and transmission of information in the brain are dependent on the interplay between structural and dynamical properties. Implicit in any pursuit aimed at understanding neural dynamics from appropriate sets of mathematically…
A line of work on Transformer-based language models such as BERT has attempted to use syntactic inductive bias to enhance the pretraining process, on the theory that building syntactic structure into the training process should reduce the…
Pinning particles at the equilibrium configuration of the liquid is expected not to affect the structure and any property that depends on the structure while slowing down the dynamics. This leads to a breakdown of the structure dynamics…
Multilingual language models (LMs) organize representations for typologically and orthographically diverse languages into a shared parameter space, yet the nature of this internal organization remains elusive. In this work, we investigate…
The microscopic dependence of glassy equilibration on sample history, both the initial configuration and the specific sequence of random noise, is examined. The temporal evolution of spin configurations in a two-dimensional Ising spin glass…