Related papers: Spin Glass Models of Syntax and Language Evolution
The paper presents a language model that develops syntactic structure and uses it to extract meaningful information from the word history, thus enabling the use of long distance dependencies. The model assigns probability to every joint…
In a recent paper we found strong evidence from simulations that the Isingantiferromagnet on ``thin'' random graphs - Feynman diagrams - displayed amean-field spin glass transition. The intrinsic interest of considering such random graphs…
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics…
I propose a novel Ising-like model of language evolution. In a simple way, Ising-like models represent the countervailing tendencies towards convergence and change present in language evolution. In the ordinary Ising-model, a node on a…
Syntactic structures used to play a vital role in natural language processing (NLP), but since the deep learning revolution, NLP has been gradually dominated by neural models that do not consider syntactic structures in their design. One…
Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the models accomplish this behavior, this study applies causal…
Distributional semantic models learn vector representations of words through the contexts they occur in. Although the choice of context (which often takes the form of a sliding window) has a direct influence on the resulting embeddings, the…
Understanding which information is encoded in deep models of spoken and written language has been the focus of much research in recent years, as it is crucial for debugging and improving these architectures. Most previous work has focused…
Syntax is fundamental to our thinking about language. Failing to capture the structure of input language could lead to generalization problems and over-parametrization. In the present work, we propose a new syntax-aware language model:…
We present results of numerical simulations on a one-dimensional Ising spin glass with long-range interactions. Parameters of the model are chosen such that it is a proxy for a short-range spin glass above the upper critical dimension (i.e.…
The neutral theory of genetic and linguistic evolution holds that the relative frequencies of variants evolve by random drift. Neutral evolution remains a plausible null model of language change. In this paper we provide evidence against…
The microscopic dependence of glassy equilibration on sample history, both the initial configuration and the specific sequence of random noise, is examined. The temporal evolution of spin configurations in a two-dimensional Ising spin glass…
We solve the q-state Potts model with anti-ferromagnetic interactions on large random lattices of finite coordination. Due to the frustration induced by the large loops and to the local tree-like structure of the lattice this model behaves…
A language dynamics model on a square lattice, which is an extension of the one popularized by Abrams and Strogatz [1], is analyzed using ODE bifurcation theory. For this model we are interested in the existence and spectral stability of…
Syntax is a latent hierarchical structure which underpins the robust and compositional nature of human language. In this work, we explore the hypothesis that syntactic dependencies can be represented in language model attention…
Synonyms and homonyms appear in all natural languages. We analyse their evolution within the framework of the signaling game. Agents in our model use reinforcement learning, where probabilities of selection of a communicated word or of its…
The processes leading to change in languages are manifold. In order to reduce ambiguity in the transmission of information, agreement on a set of conventions for recurring problems is favored. In addition to that, speakers tend to use…
We report on a refined version of our spin-glass type approach to the low-temperature physics of structural glasses. Its key idea is based on a Born von Karman expansion of the interaction potential about a set of reference positions in…
The Ising spin-glass model on the three-dimensional (d=3) hierarchical lattice with long-range ferromagnetic or spin-glass interactions is studied by the exact renormalization-group solution of the hierarchical lattice. The chaotic…
Syntactic language models (SLMs) enhance Transformers by incorporating syntactic biases through the modeling of linearized syntactic parse trees alongside surface sentences. This paper focuses on compositional SLMs that are based on…