Related papers: Three-Phase Transformer
We present ITQ3_S (Interleaved Ternary Quantization -- Specialized), a novel 3-bit weight quantization format for LLMs integrating TurboQuant (TQ), a rotation-domain strategy based on the Fast Walsh-Hadamard Transform (FWHT). Conventional…
Our aim was to enhance visual quality and quantitative accuracy of dynamic positron emission tomography (PET)uptake images by improved image reconstruction, using sophisticated sparse penalty models that incorporate both 2D spatial+1D…
Quantum kernel methods are promising for near-term quantum ma- chine learning, yet their behavior under data corruption remains insuf- ficiently understood. We analyze how quantum feature constructions degrade under controlled additive…
Accurate and efficient estimation of Channel State Information (CSI) is critical for next-generation wireless systems operating under non-stationary conditions, where user mobility, Doppler spread, and multipath dynamics rapidly alter…
Optimization of design parameters of a transformer is a crucial task to increase efficiency and lower the material cost. This research presents an approach to model a three-phase transformer and optimize design parameters to minimize the…
Recent work has made clear that the residual pathway is not mere optimization plumbing; it is part of the model's representational machinery. We agree, but argue that the cleanest way to organize this design space is through a two-axis view…
In this paper, we propose a set of transform-based neural network layers as an alternative to the $3\times3$ Conv2D layers in Convolutional Neural Networks (CNNs). The proposed layers can be implemented based on orthogonal transforms such…
It is well-known that the Haldane phase of one-dimensional spin-1 chain is a symmetry protected topological (SPT) phase, which is described by a nonlinear Sigma model (NLSM) with a Theta-term at Theta = 2Pi. In this work we study a three…
The exploration of topological phase transition (TPT) mechanisms constitutes a central theme in quantum materials research. Conventionally, transitions between Weyl semimetals (WSMs) and other topological states rely on the breaking of…
This paper reveals a novel linear characteristic exclusive to transformer decoders, including models such as GPT, LLaMA, OPT, BLOOM and others. We analyze embedding transformations between sequential layers, uncovering a near-perfect linear…
Transformers have recently shown superior performances on various vision tasks. The large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. Nevertheless, simply…
Transformers enable powerful content-based global routing via self-attention, but they lack an explicit local geometric prior along the sequence axis. As a result, the placement of locality-inducing modules in hybrid architectures has…
We study gapless phases in (3+1)d in the presence of 1-form and non-invertible duality symmetries. Using the Symmetry Topological Field Theory (SymTFT) approach, we classify the gapless symmetry-protected (gSPT) phases in these setups, with…
This paper presents Volumetric Transformer Pose estimator (VTP), the first 3D volumetric transformer framework for multi-view multi-person 3D human pose estimation. VTP aggregates features from 2D keypoints in all camera views and directly…
Recent advancements in attention mechanisms have replaced recurrent neural networks and its variants for machine translation tasks. Transformer using attention mechanism solely achieved state-of-the-art results in sequence modeling. Neural…
The Transformer and its variants have been proven to be efficient sequence learners in many different domains. Despite their staggering success, a critical issue has been the enormous number of parameters that must be trained (ranging from…
We study reduced-order models of three-dimensional perturbations in linearized channel flow using balanced proper orthogonal decomposition (BPOD). The models are obtained from three-dimensional simulations in physical space as opposed to…
Recently, learned image compression methods have made remarkable achievements, some of which have outperformed the traditional image codec VVC. The advantages of learned image compression methods over traditional image codecs can be largely…
The Transformer architecture is widely used for machine translation tasks. However, its resource-intensive nature makes it challenging to implement on constrained embedded devices, particularly where available hardware resources can vary at…
We propose a diagnostic tool for detecting non-trivial symmetry protected topological (SPT) phases protected by a symmetry group $G$ in 2+1 dimensions. Our method is based on directly studying the 1+1-dimensional anomalous edge conformal…