支持标记、稳定裕度及面向鲁棒大语言模型的新基石

自注意力通常被描述为一种灵活、内容自适应的方式，用于将 token 与其过去信息中的信息进行混合。我们在概率框架下重新诠释因果自注意力 transformer——现代基础模型的核心架构，方法类似于经典 PCA 向概率 PCA 的扩展。这种重新表述揭示了其内在变量变换的关键结构性后果：一种关于自注意力参数的障碍约束随之出现。 resulting geometry exposes a degeneracy boundary where the attention-induced mapping becomes locally ill-conditioned, yielding a stability-margin interpretation analogous to the margin in support vector machines. This, in turn, naturally gives rise to the concept of support tokens. We further show that causal transformers define a consistent stochastic process over infinite token sequences, providing a rigorous probabilistic foundation for sequence modeling. Building on this view, we derive a Bayesian MAP training objective that requires only a minimal modification to standard LLM training: adding a smooth log-barrier penalty to the usual cross-entropy loss. Empirically, the resulting training objective improves robustness to input perturbations and sharpens the margin geometry of the learned representations without sacrificing out-of-sample accuracy.