Related papers: Clustering in Deep Stochastic Transformers

We consider a generic class of gene circuits affected by nonlinear extrinsic noise. To address this nonlinearity we introduce a general perturbative methodology based on assuming timescale separation between noise and genes dynamics, with…

First return maps of interspike intervals for biological neurons that generate repetitive bursts of impulses can display stereotyped structures (neuronal signatures). Such structures have been linked to the possibility of multicoding and…

Learned optimizers -- neural networks that are trained to act as optimizers -- have the potential to dramatically accelerate training of machine learning models. However, even when meta-trained across thousands of tasks at huge…

Is it always necessary to compute tokens from shallow to deep layers in Transformers? The continued success of vanilla Transformers and their variants suggests an undoubted "yes". In this work, however, we attempt to break the depth-ordered…

Noise in entangled quantum systems is difficult to characterize due to many-body effects involving multiple degrees of freedom. This noise poses a challenge to quantum computing, where two-qubit gate performance is critical. Here, we…

Nonlinear dynamical systems exposed to changing forcing can exhibit catastrophic transitions between alternative and often markedly different states. The phenomenon of critical slowing down (CSD) can be used to anticipate such transitions…

We introduce Noise Injection Node Regularization (NINR), a method of injecting structured noise into Deep Neural Networks (DNN) during the training stage, resulting in an emergent regularizing effect. We present theoretical and empirical…

We consider entanglement-based quantum networks, where multipartite entangled resource states are distributed and stored among the nodes and locally manipulated upon request to establish the desired target configuration. Separating the…

Neural-network models of high-level brain functions such as memory recall and reasoning often rely on the presence of stochasticity. The majority of these models assumes that each neuron in the functional network is equipped with its own…

Transformers for language modeling usually rely on deterministic internal computation, with uncertainty expressed mainly at the output layer. We introduce variational neurons into Transformer feed-forward computation so that uncertainty…

We examine the role of memorization in deep learning, drawing connections to capacity, generalization, and adversarial robustness. While deep networks are capable of memorizing noise data, our results suggest that they tend to prioritize…

Transformers achieve state-of-the-art accuracy and robustness across many tasks, but an understanding of their inductive biases and how those biases differ from other neural network architectures remains elusive. In this work, we identify…

Modeling future traffic conditions often relies heavily on complex spatial-temporal neural networks to capture spatial and temporal correlations, which can overlook the inherent noise in the data. This noise, often manifesting as unexpected…

Biological and artificial neural networks develop internal representations that enable them to perform complex tasks. In artificial networks, the effectiveness of these models relies on their ability to build task specific representation, a…

Recent studies indicate that deep neural networks degrade in generalization performance under noisy supervision. Existing methods focus on isolating clean subsets or correcting noisy labels, facing limitations such as high computational…

Transformers have achieved remarkable success in several domains, ranging from natural language processing to computer vision. Nevertheless, it has been recently shown that stacking self-attention layers - the distinctive architectural…

Clustering is essential to many tasks in pattern recognition and computer vision. With the advent of deep learning, there is an increasing interest in learning deep unsupervised representations for clustering analysis. Many works on this…

Transformer-based models have achieved remarkable success across a wide range of domains, yet our understanding of their training dynamics remains limited. In this work, we identify a recurrent focus-dilution cycle in attention learning and…

We study the emergence of anticoncentration and approximate unitary design behavior in local Brownian circuits. The dynamics of circuit averaged moments of the probability distribution and entropies of the output state can be represented as…

We investigate the transport properties of particles driven by colored noise in a flashing ratchet potential, focusing on both non-interacting and single-file interacting regimes. The model incorporates memory effects via a non-Markovian…