Related papers: Three-Phase Transformer
Symmetry-protected topological (SPT) phases are short-range entangled quantum phases with symmetry, which have gapped excitations in the bulk and gapless modes at the edge. In this paper, we study the SPT phases in the spin-1 Heisenberg…
The wave-appropriate reconstruction approach decomposes the reconstruction procedure into characteristic wave families, centralizing non-acoustic waves to minimize dissipation while retaining an upwind bias for acoustic waves. In previous…
The hole doped Si(111)(2root3x2root3)R30(degrees)-Sn interface exhibits a symmetry-breaking insulator-insulator transition below 100 K that appears to be triggered by electron tunneling into the empty surface-state bands. No such transition…
Transformers trained in low precision can suffer forward-error amplification. We give a first-order, module-wise theory that predicts when and where errors grow. For self-attention we derive a per-layer bound that factorizes into three…
We propose a three-module extension of score-based VAMP (SC-VAMP) for signal recovery in nonlinear channels, where the received signal is obtained by applying a nonlinearity to a linear mixture of the transmitted signal, followed by…
In recent years, Transformer has achieved good results in Natural Language Processing (NLP) and has also started to expand into Computer Vision (CV). Excellent models such as the Vision Transformer and Swin Transformer have emerged. At the…
State-of-the-art computer- and robot-assisted surgery systems heavily depend on intraoperative imaging technologies such as CT and fluoroscopy to generate detailed 3D visualization of the patient's anatomy. While imaging techniques are…
Time series forecasting is essential for a wide range of real-world applications. Recent studies have shown the superiority of Transformer in dealing with such problems, especially long sequence time series input(LSTI) and long sequence…
We present a family of local quantum channels whose steady-states exhibit stable mixed-state symmetry-protected topological (SPT) order. Motivated by recent experimental progress on "erasure conversion" techniques that allow one to identify…
We theoretically investigate whether the Query, Key, Value weight triplet can be reduced in encoder-only and decoder-only transformers. Under mild assumptions, we prove that one of the Query, Key or Value weights are redundant and can be…
Fine-tuning adapts pretrained networks to new objectives. Whether the resulting depth profile of representational change reflects an intrinsic property of the model or the magnitude of gradient flow has not been tested directly. We measure…
This work, based on Random Matrix Theory (RMT), introduces a novel early-stopping strategy for Transformer training dynamics. Utilizing the Power Law (PL) fit to tansformer attention matrices as a probe, we demarcate training into three…
Mechanistic interpretability seeks to reverse engineer a trained neural network by identifying the minimal subset of internal components. We perform a mechanistic interpretability analysis of the Particle Transformer architecture, trained…
What is the computational model behind a Transformer? Where recurrent neural networks have direct parallels in finite state machines, allowing clear discussion and thought around architecture variants or trained models, Transformers have no…
The dot product self-attention (DPSA) is a fundamental component of transformers. However, scaling them to long sequences, like documents or high-resolution images, becomes prohibitively expensive due to quadratic time and memory…
We introduce the Convolutional Set Transformer (CST), a novel neural architecture designed to process image sets of arbitrary cardinality that are visually heterogeneous yet share high-level semantics - such as a common category, scene, or…
Existing semantic segmentation works have been mainly focused on designing effective decoders; however, the computational load introduced by the overall structure has long been ignored, which hinders their applications on…
While Transformers have demonstrated remarkable potential in modeling Partial Differential Equations (PDEs), modeling large-scale unstructured meshes with complex geometries remains a significant challenge. Existing efficient architectures…
Topological phase transition induced by spatially-tuned single-site measurement is investigated in a measurement-only circuit, in which three different types of projective measurement operator. Specific spatial setting and combination of…
The conception of the conformal phase transiton (CPT), which is relevant for the description of non-perturbative dynamics in gauge theories, is introduced and elaborated. The main features of such a phase transition are established. In…