Related papers: Integrated electro-optic attention nonlinearities …
The attention mechanism in Transformers is an important primitive for accurate and scalable sequence modeling. Its quadratic-compute and linear-memory complexity however remain significant bottlenecks. Linear attention and state-space…
Transformer networks have emerged as the state-of-the-art approach for natural language processing tasks and are gaining popularity in other domains such as computer vision and audio processing. However, the efficient hardware acceleration…
Attention is a core component of transformer architecture, whether encoder-only, decoder-only, or encoder-decoder model. However, the standard softmax attention often produces noisy probability distribution, which can impair effective…
Since its introduction, softmax attention has become the backbone of modern transformer architectures due to its expressiveness and scalability across a wide range of tasks. However, the main drawback of softmax attention is the quadratic…
Linear attention methods offer Transformers $O(N)$ complexity but typically underperform standard softmax attention. We identify two fundamental limitations affecting these approaches: the restriction to convex combinations that only…
Integrated thin-film lithium niobate (LN) electro-optic (EO) modulators of broad bandwidth, low insertion loss, low cost and high production rate are essential elements in contemporary inter-connection industries and disruptive…
Transformer models typically calculate attention matrices using dot products, which have limitations when capturing nonlinear relationships between embedding vectors. We propose Neural Attention, a technique that replaces dot products with…
We present Lightning Attention, the first linear attention implementation that maintains a constant training speed for various sequence lengths under fixed memory consumption. Due to the issue with cumulative summation operations (cumsum),…
Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also known for their significant computational and storage…
Many applications of metasurfaces require an ability to dynamically change their properties in time domain. Electrical tuning techniques are of particular interest, since they pave a way to on-chip integration of metasurfaces with…
Vision transformers have shown great success on numerous computer vision tasks. However, its central component, softmax attention, prohibits vision transformers from scaling up to high-resolution images, due to both the computational…
Here we show a photonic computing accelerator utilizing a system-level thin-film lithium niobate circuit which overcomes this limitation. Leveraging the strong electro-optic (Pockels) effect and the scalability of this platform, we…
Looped Transformers (LT) have emerged as a powerful architecture by iterating their layers multiple times before decoding the final token. However, pairing them with full attention retains quadratic complexity, making them computationally…
Nonlinear frequency conversion offers powerful capabilities for applications in telecommunications, signal processing, and computing. Thin-film lithium niobate (TFLN) has emerged as a promising integrated photonics platform due to its…
Linear RNNs with gating recently demonstrated competitive performance compared to Transformers in language modeling. Although their linear compute scaling in sequence length offers theoretical runtime advantages over Transformers, realizing…
Periodically poled thin-film lithium niobate (TFLN) waveguides, which enable efficient quadratic nonlinear processes, serve as crucial foundation for classical and quantum signal processing. To expand their application scope, we provide the…
Vision Transformers (ViTs) have recently taken computer vision by storm. However, the softmax attention underlying ViTs comes with a quadratic complexity in time and memory, hindering the application of ViTs to high-resolution images. We…
High-performance electro-optic modulators play a critical role in modern telecommunication networks and intra-datacenter interconnects. Low driving voltage, large electro-optic bandwidth, compact device size, and multi-band operation…
Periodically poled thin-film lithium niobate (TFLN) waveguides have emerged as a leading platform for highly efficient frequency conversion in the near-infrared. However, the commonly used silica bottom-cladding results in high absorption…
Transformer has shown great successes in natural language processing, computer vision, and audio processing. As one of its core components, the softmax attention helps to capture long-range dependencies yet prohibits its scale-up due to the…