A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization
Abstract
Conventional regularization is designed to control variance, but in small-data regression it can also aggravate underfitting when predictive signal is concentrated in weak directions of a restricted representation. We study a negative-capable ridge family that permits a feasible negative region whenever the estimator remains well posed, and show that negative regularization acts there as controlled anti-shrinkage by increasing effective complexity most strongly along weak eigendirections. Building on this mechanism, we formalize weak-spectrum underfitting, derive a sign-switch result under conservative baseline shrinkage, and study criterion-based automatic selection over the full negative-capable family. Synthetic and semi-synthetic experiments support the theory by verifying feasibility, spectral complexity increase, sign-switch behavior, and effective recovery of negative adjustments in the predicted regimes.
Keywords
Cite
@article{arxiv.2508.17412,
title = {A Ridge Too Far: Correcting Over-Shrinkage via Negative Regularization},
author = {Dongseok Kim and Gisung Oh},
journal= {arXiv preprint arXiv:2508.17412},
year = {2026}
}
Comments
Substantially revised and reorganized version with a new title, updated framing, and new experiments; the core idea of the work remains unchanged