Related papers: Clustering in Deep Stochastic Transformers

Randomized smoothing is a well-established method for achieving certified robustness against l2-adversarial perturbations. By incorporating a denoiser before the base classifier, pretrained classifiers can be seamlessly integrated into…

Several works have shown that the regularization mechanisms underlying deep neural networks' generalization performances are still poorly understood. In this paper, we hypothesize that deep neural networks are regularized through their…

It has been shown that injecting noise into the neural network weights during the training process leads to a better generalization of the resulting model. Noise injection in the distributed setup is a straightforward technique and it…

Transformers have exhibited exceptional capabilities in sequence modeling tasks, leveraging self-attention and in-context learning. Critical to this success are induction heads, attention circuits that enable copying tokens based on their…

We study stochastic optimization from a joint continuous-discrete point of view. Starting from a second-order stochastic differential equation interpreted as a noisy accelerated gradient flow, we discretize the dynamics by a fully implicit…

Prediction via deterministic continuous-time models will always be subject to model error, for example due to unexplainable phenomena, uncertainties in any data driving the model, or discretisation/resolution issues. In this paper, we build…

Since its introduction in 2017, Transformer has emerged as the leading neural network architecture, catalyzing revolutionary advancements in many AI disciplines. The key innovation in Transformer is a Self-Attention (SA) mechanism designed…

Biological systems operate under persistent noise, which can alter system states and induce transitions between attractors. Here, we study the attractor dynamics of Boolean networks focusing on the transitions between attractors induced by…

Current quantum hardware is subject to various sources of noise that limits the access to multi-qubit entangled states. Quantum autoencoder circuits with a single qubit bottleneck have shown capability to correct error in noisy entangled…

The process of stochastic Turing instability on a network is discussed for a specific case study, the stochastic Brusselator model. The system is shown to spontaneously differentiate into activator-rich and activator-poor nodes, outside the…

A widely cited result by Dong et al. (2021) showed that Transformers built from self-attention alone, without skip connections or feed-forward layers, suffer from rapid rank collapse: all token representations converge to a single…

Controlling dopant transport with high spatial precision is crucial for improving the semiconductor functionality, reliability, and scalability. Although prior models of noise-assisted diffusion have been largely confined to idealized one…

Standard transformer architectures apply the same number of layers to every token regardless of contextual difficulty. We present Token-Selective Attention (TSA), a learned per-token gate on residual updates between consecutive transformer…

Solar spectropolarimetric inversion -- inferring atmospheric conditions from the Stokes vector -- is a key diagnostic tool for understanding solar magnetism, but traditional inversion methods are computationally expensive and sensitive to…

Deep neural networks have been shown to be very powerful methods for many supervised learning tasks. However, they can also easily overfit to training set biases, i.e., label noise and class imbalance. While both learning with noisy labels…

Encoder transformer models compress information from all tokens in a sequence into a single [CLS] token to represent global context. This approach risks diluting fine-grained or hierarchical features, leading to information loss in…

Many approaches to modelling reaction-diffusion systems with anomalous transport rely on deterministic equations and ignore fluctuations arising due to finite particle numbers. Starting from an individual-based model we use a…

In this paper we present a framework for investigating coloured noise in reaction-diffusion systems. We start by considering a deterministic reaction-diffusion equation and show how external forcing can cause temporally correlated or…

We study the effects of noise on the dynamics of a system of coupled self-propelling particles in the case where the coupling is time-delayed, and the delays are discrete and randomly generated. Previous work has demonstrated that the…

Intrinsic or demographic noise has been shown to play an important role in the dynamics of a variety of systems including predator-prey populations, intracellular biochemical reactions, and oscillatory chemical reaction systems, and is…