Related papers: Spin Glass Models of Syntax and Language Evolution
This paper proposes a novel approach to word embeddings in Transformer models by utilizing spinors from geometric algebra. Spinors offer a rich mathematical framework capable of capturing complex relationships and transformations in…
Syntax is usually studied in the realm of linguistics and refers to the arrangement of words in a sentence. Similarly, an image can be considered as a visual 'sentence', with the semantic parts of the image acting as 'words'. While visual…
The glassy behavior observed in the pyrochlore magnet Y2Mo2O7, where the magnetic Mo^{4+} ions interact predominantly via isotropic nearest neighbor antiferromagnetic exchange, possibly with additional weak disorder, is a distinct class of…
We deploy the methods of controlled psycholinguistic experimentation to shed light on the extent to which the behavior of neural network language models reflects incremental representations of syntactic state. To do so, we examine model…
Incorporating stronger syntactic biases into neural language models (LMs) is a long-standing goal, but research in this area often focuses on modeling English text, where constituent treebanks are readily available. Extending constituent…
The field of spoken language processing is undergoing a shift from training custom-built, task-specific models toward using and optimizing spoken language models (SLMs) which act as universal speech processing systems. This trend is similar…
This paper develops a general data-driven approach to stochastic elastoplastic modelling that leverages atomistic simulation data directly rather than by fitting parameters. The approach is developed in the context of metallic glasses,…
Construction Grammar (CxG) is a paradigm from cognitive linguistics emphasising the connection between syntax and semantics. Rather than rules that operate on lexical items, it posits constructions as the central building blocks of…
Self-modulating mechanisms introduce dynamic adaptation capabilities within language models through contextual realignment strategies that influence token embedding trajectories across extended sequences. Contextual Flux is explored as an…
We recently showed that the two-dimensional Ising spin glass allows for a line of renormalization group fixed points which explains properties observed in numerical studies. We observe that this exact result corresponds to enhancement to a…
Methods and insights from statistical physics are finding an increasing variety of applications where one seeks to understand the emergent properties of a complex interacting system. One such area concerns the dynamics of language at a…
Understanding what and how causal dynamical mechanisms generate collective phenomena is a central challenge in complexity science. Recent studies have focused on identifying the mechanisms underlying the synergistic interdependencies that…
It is commonly believed that knowledge of syntactic structure should improve language modeling. However, effectively and computationally efficiently incorporating syntactic structure into neural language models has been a challenging topic.…
We study a p-spin spin-glass model to understand if the finite-temperature glass transition found in the mean-field regime of p-spin models, and used to model the behavior of structural glasses, persists in the non-mean-field regime. By…
This review presents various aspects of a mean-field spin glass model known as the p-spin spherical spin glass model, which has raised a lot of interest in the study of spin glasses, and also for its possible links with a mean-field theory…
While vector-based language representations from pretrained language models have set a new standard for many NLP tasks, there is not yet a complete accounting of their inner workings. In particular, it is not entirely clear what aspects of…
Human communication systems, such as language, evolve culturally; their components undergo reproduction and variation. However, a role for selection in cultural evolutionary dynamics is less clear. Often neutral evolution (also known as…
Languages vary considerably in syntactic structure. About 40% of the world's languages have subject-verb-object order, and about 40% have subject-object-verb order. Extensive work has sought to explain this word order variation across…
Inspired by striking advances in language modeling, there has recently been much interest in developing autogressive sequence models that are amenable to analytical study. In this short note, we consider extensions of simple disordered…
The theoretical code-switching (CS) literature provides numerous pointwise investigations that aim to explain patterns in CS, i.e. why bilinguals switch language in certain positions in a sentence more often than in others. A resulting…