English

Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers

Machine Learning 2025-12-30 v1 Artificial Intelligence

Abstract

We propose a novel framework, Continuous_Time Attention, which infuses partial differential equations (PDEs) into the Transformer's attention mechanism to address the challenges of extremely long input sequences. Instead of relying solely on a static attention matrix, we allow attention weights to evolve over a pseudo_time dimension via diffusion, wave, or reaction_diffusion dynamics. This mechanism systematically smooths local noise, enhances long_range dependencies, and stabilizes gradient flow. Theoretically, our analysis shows that PDE_based attention leads to better optimization landscapes and polynomial rather than exponential decay of distant interactions. Empirically, we benchmark our method on diverse experiments_demonstrating consistent gains over both standard and specialized long sequence Transformer variants. Our findings highlight the potential of PDE_based formulations to enrich attention mechanisms with continuous_time dynamics and global coherence.

Keywords

Cite

@article{arxiv.2505.20666,
  title  = {Continuous-Time Attention: PDE-Guided Mechanisms for Long-Sequence Transformers},
  author = {Yukun Zhang and Xueqing Zhou},
  journal= {arXiv preprint arXiv:2505.20666},
  year   = {2025}
}
R2 v1 2026-07-01T02:41:29.720Z