Related papers: Three-Phase Transformer
While convolutional neural networks have shown a tremendous impact on various computer vision tasks, they generally demonstrate limitations in explicitly modeling long-range dependencies due to the intrinsic locality of the convolution…
We present a three-step recipe for identifying attention-head circuits in pretrained transformers. A per-head spectral signal -- the time-integrated participation ratio of each head's attention output -- ranks heads doing sustained…
We review the electronic structure of nickelate superconductors with and without effects of electronic correlations. As a minimal model we identify the one-band Hubbard model for the Ni 3$d_{x^2-y^2}$ orbital plus a pocket around the…
In this work, we simulate the expected device performance and the scaling perspectives of Carbon nanotube Field Effect Transistors (CNT-FETs), with doped source and drain extensions. The simulations are based on the self-consistent solution…
Transformers have emerged as viable alternatives to convolutional neural networks owing to their ability to learn non-local region relationships in the spatial domain. The self-attention mechanism of the transformer enables transformers to…
Recently, image restoration transformers have achieved comparable performance with previous state-of-the-art CNNs. However, how to efficiently leverage such architectures remains an open problem. In this work, we present Dual-former whose…
We study the phase structure and charge transport at finite temperature and chemical potential in the non-Hermitian PT-symmetric holographic model of arXiv:1912.06647. The non-Hermitian PT-symmetric deformation is realized by promoting the…
We have tried to interpret the physical role of the three-tangle and $\pi$-tangle in the real physical information process. For the model calculation we adopt the three-party teleportation scheme through the various noisy channels. The…
Designing an efficient yet deployment-friendly 3D backbone to handle sparse point clouds is a fundamental problem in 3D perception. Compared with the customized sparse convolution, the attention mechanism in Transformers is more appropriate…
Symmetry-protected topological (SPT) phases are commonly required to have an energy gap, but recent work has extended the concept to gapless settings. This raises a natural question: what happens at transitions between inequivalent gapless…
As the computing power of modern hardware is increasing strongly, pre-trained deep learning models (e.g., BERT, GPT-3) learned on large-scale datasets have shown their effectiveness over conventional methods. The big progress is mainly…
This paper presents stable, radix-2, completely recursive discrete cosine transformation algorithms DCT-I and DCT-III solely based on DCT-I, DCT-II, DCT-III, and DCT-IV having sparse and orthogonal factors. Error bounds for computing the…
We study Latent Recurrent Transformer (LRT), a lightweight augmentation of autoregressive transformers that reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token. Because this source…
The pi N --> eta N and eta N --> eta N partial wave T-matrices for the eight lowest partial waves have been obtained in a three coupled channel model with unitarity manifestly imposed. The two physical channels are pi_N and eta_N, and the…
We report charge transport measurements in nanoscale vertical pillar structures incorporating ultrathin layers of the organic semiconductor poly(3-hexylthiophene)(P3HT). P3HT layers with thickness down to 5 nm are gently top-contacted using…
Image Captioning is an important Language and Vision task that finds application in a variety of contexts, ranging from healthcare to autonomous vehicles. As many real-world applications rely on devices with limited resources, much effort…
We introduce directional routing, a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router, at 3.9% parameter cost. We train a 433M-parameter model alongside an…
Neural shape representation generally refers to representing 3D geometry using neural networks, e.g., computing a signed distance or occupancy value at a specific spatial position. In this paper we present a neural-network architecture…
Effectively preserving and encoding structure features from objects in irregular and sparse LiDAR points is a key challenge to 3D object detection on point cloud. Recently, Transformer has demonstrated promising performance on many 2D and…
Transformer has been applied in the field of computer vision due to its excellent performance in natural language processing, surpassing traditional convolutional neural networks and achieving new state-of-the-art. ViT divides an image into…