Related papers: Attention to Order: Transformers Discover Phase Tr…
We employ unsupervised machine learning techniques to learn latent parameters which best describe states of the two-dimensional Ising model and the three-dimensional XY model. These methods range from principal component analysis to…
Transformer is a ubiquitous model for natural language processing and has attracted wide attentions in computer vision. The attention maps are indispensable for a transformer model to encode the dependencies among input tokens. However,…
This paper introduces a high-order Markov chain task to investigate how transformers learn to integrate information from multiple past positions with varying statistical significance. We demonstrate that transformers learn this task…
Machine-learning driven models have proven to be powerful tools for the identification of phases of matter. In particular, unsupervised methods hold the promise to help discover new phases of matter without the need for any prior…
Unsupervised learning is a discipline of machine learning which aims at discovering patterns in big data sets or classifying the data into several categories without being trained explicitly. We show that unsupervised learning techniques…
Neural networks can be used to identify phases and phase transitions in condensed matter systems via supervised machine learning. Readily programmable through modern software libraries, we show that a standard feed-forward neural network…
Transformer networks have seen great success in natural language processing and machine vision, where task objectives such as next word prediction and image classification benefit from nuanced context sensitivity across high-dimensional…
With the attention mechanism, transformers achieve significant empirical successes. Despite the intuitive understanding that transformers perform relational inference over long sequences to produce desirable representations, we lack a…
Memory is fundamental to intelligence, enabling learning, reasoning, and adaptability across biological and artificial systems. While Transformer architectures excel at sequence modeling, they face critical limitations in long-range context…
Transformers have demonstrated remarkable success across various applications. However, the success of transformers have not been understood in theory. In this work, we give a case study of how transformers can be trained to learn a classic…
Machine learning-inspired techniques have emerged as a new paradigm for analysis of phase transitions in quantum matter. In this work, we introduce a supervised learning algorithm for studying critical phenomena from measurement data, which…
Still under debate is the question of whether machine learning is capable of going beyond black-box modeling for complex physical systems. We investigate the generalizing and interpretability properties of learning algorithms. To this end,…
Recent years have witnessed a growing interest in using machine learning to predict and identify phase transitions in various systems. Here we adopt convolutional neural networks (CNNs) to study the phase transitions of Vicsek model,…
Transformers have recently revolutionized many domains in modern machine learning and one salient discovery is their remarkable in-context learning capability, where models can solve an unseen task by utilizing task-specific prompts without…
There is a growing interest in the ability of neural networks to execute algorithmic tasks (e.g., arithmetic, summary statistics, and sorting). The goal of this work is to better understand the role of attention in Transformers for…
Although transformer-based models have shown exceptional empirical performance, the fundamental principles governing their training dynamics are inadequately characterized beyond configuration-specific studies. Inspired by empirical…
Supervised Learning has been successfully used to produce phase diagrams and identify phase boundaries when local order parameters are unavailable. Here, we apply unsupervised learning to this task. By using readily available clustering…
The Transformer is a highly successful deep learning model that has revolutionised the world of artificial neural networks, first in natural language processing and later in computer vision. This model is based on the attention mechanism…
Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies within temporal…
We study two-dimensional Ising spins, evolving through reinforcement learning using their state, action, and reward. The state of a spin is defined as whether it is in the majority or minority with its nearest neighbours. The spin updates…