We introduce softpick, a rectified, not sum-to-one, drop-in replacement for softmax in transformer attention mechanisms that eliminates attention sink and massive activations. Our experiments with 340M and 1.8B parameter models demonstrate that softpick achieves 0\% sink rate consistently. The softpick transformers produce hidden states with significantly lower kurtosis and creates sparse attention maps. Quantized models using softpick outperform softmax on standard benchmarks, with a particularly pronounced advantage at lower bit precisions. Our analysis and discussion shows how softpick has the potential to open new possibilities for quantization, low-precision training, sparsity optimization, pruning, and interpretability. Our code: https://github.com/zaydzuhri/softpick-attention
Cite
@article{arxiv.2504.20966,
title = {Softpick: No Attention Sink, No Massive Activations with Rectified Softmax},
author = {Zayd M. K. Zuhri and Erland Hilman Fuadi and Alham Fikri Aji},
journal= {arXiv preprint arXiv:2504.20966},
year = {2026}
}