Related papers: Clustering in Deep Stochastic Transformers
Transformers serve as the foundational architecture for large language and video generation models, such as GPT, BERT, SORA and their successors. Empirical studies have demonstrated that real-world data and learning tasks exhibit…
Initialization of composite quantum systems into highly entangled states is usually a must to allow their use for quantum technologies. However, the presence of unavoidable noise in the preparation stage makes the system state mixed, thus…
Although classifying topological quantum phases have attracted great interests, the absence of local order parameter generically makes it challenging to detect a topological phase transition from experimental data. Recent advances in…
We address a learning-based quantum error mitigation method, which utilizes deep neural network applied at the postprocessing stage, and study its performance in presence of different types of quantum noises. We concentrate on the…
Autoencoders have emerged as a useful framework for unsupervised learning of internal representations, and a wide variety of apparently conceptually disparate regularization techniques have been proposed to generate useful features. Here we…
The diffusion-driven Turing instability is a potential mechanism for spatial pattern formation in numerous biological and chemical systems. However, engineering these patterns and demonstrating that they are produced by this mechanism is…
In label-noise learning, estimating the transition matrix has attracted more and more attention as the matrix plays an important role in building statistically consistent classifiers. However, it is very challenging to estimate the…
Recurrent networks of dynamic elements frequently exhibit emergent collective oscillations, which can display substantial regularity even when the individual elements are considerably noisy. How noise-induced dynamics at the local level…
Demographic noise has profound effects on evolutionary and population dynamics, as well as on chemical reaction systems and models of epidemiology. Such noise is intrinsic and due to the discreteness of the dynamics in finite populations.…
Recent work has shown that Transformers can perform in-context learning for linear regression under restrictive assumptions, including i.i.d. data, Gaussian noise, and Gaussian regression coefficients. However, real-world data often violate…
This paper introduces a high-order Markov chain task to investigate how transformers learn to integrate information from multiple past positions with varying statistical significance. We demonstrate that transformers learn this task…
Transformers have transformed modern machine learning, driving breakthroughs in computer vision, natural language processing, and robotics. At the core of their success lies the attention mechanism, which enables the modeling of global…
A deterministic dynamical system that slowly passes through a generic fold-type (saddle-node) bifurcation can be reduced to one-dimensional dynamics close to the bifurcation because of the centre manifold theorem. It is often tacitly…
Equilibrium topological phases are robust against weak static disorder but may break down in the strong disorder regime. Here we explore the stability of the quench-induced emergent dynamical topology in the presence of dynamical noise. We…
Initialization of parameters in deep neural networks has been shown to have a big impact on the performance of the networks (Mishkin & Matas, 2015). The initialization scheme devised by He et al, allowed convolution activations to carry a…
Learning from multiple annotators aims to induce a high-quality classifier from training instances, where each of them is associated with a set of possibly noisy labels provided by multiple annotators under the influence of their varying…
Deep clustering is the first method to handle general audio separation scenarios with multiple sources of the same type and an arbitrary number of sources, performing impressively in speaker-independent speech separation tasks. However,…
Injecting gaussian noise into training features is well known to have regularization properties. This paper considers noise injections to numeric or categoric tabular features as passed to inference, which translates inference to a…
Stochastic regularisation is an important weapon in the arsenal of a deep learning practitioner. However, despite recent theoretical advances, our understanding of how noise influences signal propagation in deep neural networks remains…
Attention mechanism is a fundamental component of the transformer model and plays a significant role in its success. However, the theoretical understanding of how attention learns to select tokens is still an emerging area of research. In…