Related papers: QAMA: Scalable Quantum Annealing Multi-Head Attent…

The Transformer model architecture has become one of the most widely used in deep learning and the attention mechanism is at its core. The standard attention formulation uses a softmax operation applied to a scaled dot product between query…

Large language model (LLM) inference demands significant amount of computation and memory, especially in the key attention mechanism. While techniques, such as quantization and acceleration algorithms, like FlashAttention, have improved…

Vision Transformers (ViT) serve as powerful vision models. Unlike convolutional neural networks, which dominated vision research in previous years, vision transformers enjoy the ability to capture long-range dependencies in the data.…

Developing quantum algorithms adaptive to specific constraints of near-term devices is an essential step towards practical quantum advantage. In a recent work [Phys. Rev. Lett. 131, 103601(2023)], we show cold atoms in an optical cavity can…

Deploying long-context large language models (LLMs) is essential but poses significant computational and memory challenges. Caching all Key and Value (KV) states across all attention heads consumes substantial memory. Existing KV cache…

Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max…

The Multi-Head Attention mechanism is central to LLM operation, and multiple works target its compute and memory efficiency during training. While most works focus on approximating the scaled dot product, the memory consumption of the…

The Segment Anything Model (SAM) has advanced interactive segmentation but is limited by the high computational cost on high-resolution images. This requires downsampling to meet GPU constraints, sacrificing the fine-grained details needed…

With rapid advancements in machine learning, first-order algorithms have emerged as the backbone of modern optimization techniques, owing to their computational efficiency and low memory requirements. Recently, the connection between…

The quantum approximate optimization algorithm (QAOA) is a hybrid quantum-classical variational algorithm that offers the potential to handle combinatorial optimization problems. Introducing constraints in such combinatorial optimization…

Pre-trained transformer models with extended context windows are notoriously expensive to run at scale, often limiting real-world deployment due to their high computational and memory requirements. In this paper, we introduce Hamming…

Scaling inference for large language models (LLMs) is increasingly constrained by limited GPU memory, especially due to growing key-value (KV) caches required for long-context generation. While existing approaches offload KV caches to CPU…

Attention mechanism is a significant part of Transformer models. It helps extract features from embedded vectors by adding global information and its expressivity has been proved to be powerful. Nevertheless, the quadratic complexity…

The prospect of quantum solutions for complicated optimization problems is contingent on mapping the original problem onto a tractable quantum energy landscape, e.g. an Ising-type Hamiltonian. Subsequently, techniques like adiabatic…

Efficient production planning is essential in modern manufacturing to improve performance indicators such as lead time and to reduce reliance on human intuition. While mathematical optimization approaches, formulated as job shop scheduling…

A key advantage of Recurrent Neural Networks (RNNs) over Transformers is their linear computational and space complexity enables faster training and inference for long sequences. However, RNNs are fundamentally unable to randomly access…

Quantum computers may outperform classical computers on machine learning tasks. In recent years, a variety of quantum algorithms promising unparalleled potential to enhance, speed up, or innovate machine learning have been proposed. Yet,…

Quantum processors promise a paradigm shift in high-performance computing which needs to be assessed by accurate benchmarking measures. In this work, we introduce a new benchmark for variational quantum algorithm (VQA), recently proposed as…

Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference…

In the era of noisy-intermediate-scale quantum computers, we expect to see quantum devices with increasing numbers of qubits emerge in the foreseeable future. To practically run quantum programs, logical qubits have to be mapped to the…