English

Spectral Transformer Neural Processes

Machine Learning 2026-05-12 v1 Artificial Intelligence

Abstract

Time series, spatial data, and images are natural applications of Neural Processes. However, when such data exhibit strong periodicity and quasi-periodicity, existing methods often suffer from underfitting and generalise poorly beyond the training distribution. In this work, we propose Spectral Transformer Neural Processes (STNPs), a frequency-aware extension of Transformer Neural Processes (TNPs). STNPs introduce a Spectral Aggregator that estimates an empirical context spectrum, compresses it into a spectral mixture, samples task-adaptive spectral features, and concatenates them with time-domain embeddings, thereby injecting a spectral-mixture-kernel bias into TNPs. This design reshapes the similarity geometry, allowing inputs that are distant in Euclidean space to remain close in an induced periodic manifold while enhancing time-frequency interactions. Extensive experiments on synthetic regression tasks, real-world time-series datasets, and an image dataset demonstrate that STNPs consistently improve predictive performance over existing baselines, extending Neural Processes beyond translation equivariance towards effective modelling of periodicity and quasi-periodicity.

Keywords

Cite

@article{arxiv.2605.09498,
  title  = {Spectral Transformer Neural Processes},
  author = {Xianhe Chen and Hao Chen and Yingzhen Li},
  journal= {arXiv preprint arXiv:2605.09498},
  year   = {2026}
}

Comments

37 pages, 10 figures, 18 tables

R2 v1 2026-07-01T13:01:42.513Z