Our paper challenges claims from prior research that transformer-based models, when learning in context, implicitly implement standard learning algorithms. We present empirical evidence inconsistent with this view and provide a mathematical analysis demonstrating that transformers cannot achieve general predictive accuracy due to inherent architectural limitations.
@article{arxiv.2502.03503,
title = {Analyzing limits for in-context learning},
author = {Omar Naim and Jerome Bolte and Nicholas Asher},
journal= {arXiv preprint arXiv:2502.03503},
year = {2025}
}
Comments
39th Conference on Neural Information Processing Systems (NeurIPS 2025) Workshop What Can t Transformers Do?