Related papers: Clustering in Deep Stochastic Transformers

The general trend in NLP is towards increasing model capacity and performance via deeper neural networks. However, simply stacking more layers of the popular Transformer architecture for machine translation results in poor convergence and…

Partitioning large networks into stable clusters of synchronized nodes is a challenging task. Recent approaches based on spectral analysis can provide exact results on specific dynamics but remain unfeasible for very large networks.…

We investigate the effect of a two-level jump process or random telegraph noise on a square wave driven tight-binding lattice. In the absence of the noise, the system is known to exhibit dynamical localization for specific ratios of the…

Iterative clustering algorithms help us to learn the insights behind the data. Unfortunately, this may allow adversaries to infer the privacy of individuals with some background knowledge. In the worst case, the adversaries know the…

We present a representation learning method that learns features at multiple different levels of scale. Working within the unsupervised framework of denoising autoencoders, we observe that when the input is heavily corrupted during…

While diffusion models have achieved great success in generating continuous signals such as images and audio, it remains elusive for diffusion models in learning discrete sequence data like natural languages. Although recent advances…

Randomization is a powerful tool that endows algorithms with remarkable properties. For instance, randomized algorithms excel in adversarial settings, often surpassing the worst-case performance of deterministic algorithms with large…

The dynamics of noise-resilient Boolean networks with majority functions and diverse topologies is investigated. A wide class of possible topological configurations is parametrized as a stochastic blockmodel. For this class of networks, the…

We propose a new non-equilibrium model for spatial pattern formation on the basis of local information transfer. Unlike standard models of pattern formation it is not based on the Turing instability. Information is transmitted through the…

Random label noises (or observational noises) widely exist in practical machine learning settings. While previous studies primarily focus on the affects of label noises to the performance of learning, our work intends to investigate the…

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in using Markov input processes to study transformers.…

Quantum dynamics in a strongly disordered quantum many-body system show localization properties. The initial state memory is maintained owing to slow relaxation when the system is in the localized regime. This work demonstrates how…

Self-supervised training methods for transformers have demonstrated remarkable performance across various domains. Previous transformer-based models, such as masked autoencoders (MAE), typically utilize a single normalization layer for both…

A stochastic model of excitatory and inhibitory interactions which bears universality traits is introduced and studied. The endogenous component of noise, stemming from finite size corrections, drives robust inter-nodes correlations, that…

We present the numerical estimation of noise parameter induced in the dynamics of the variables by random particle interactions involved in the stochastic chemical oscillator and use it as order parameter to detect the transition from…

We analyze a class of chemical reaction networks under mass-action kinetics and involving multiple time-scales, whose deterministic and stochastic models display qualitative differences. The networks are inspired by gene-regulatory…

The Transformer translation model employs residual connection and layer normalization to ease the optimization difficulties caused by its multi-layer encoder/decoder structure. Previous research shows that even with residual connection and…

Transformer architecture has become ubiquitous in the natural language processing field. To interpret the Transformer-based models, their attention patterns have been extensively analyzed. However, the Transformer architecture is not only…

This paper proposes a novel deep subspace clustering approach which uses convolutional autoencoders to transform input images into new representations lying on a union of linear subspaces. The first contribution of our work is to insert…

We develop a microscopic transport theory in a randomly driven fermionic model with and without linear potential. The operator dynamics arise from the competition between noisy and static couplings, leading to diffusion regardless of…