MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment
Abstract
Although multi-scales representation learning enables elastic-dimension embeddings, nested subspaces often suffer from dimensional redundancy and spectral collapse. To address this, we introduce MIC, a framework that optimizes the geometric landscape of multi-granular embeddings through isotropic subspace alignment. MIC employs Soft Collapse Regularization (SCR) to mitigate redundancy between prefix and residual subspaces via cross-correlation penalties, alongside Spectral Isotropy Regularization (SIR) to ensure hyper-spherical uniformity in low-dimensional prefixes. By unifying these strategies through a self-distillation objective, MIC generates semantically dense representations that maintain high discriminative power. Our experiments demonstrate that MIC significantly outperforms standard baselines, particularly in high-compression scenarios where maintaining informational capacity is most critical.
Comments: Accepted at the GlobalSouthML Workshop at ICML 2026. 13 pages, 2 figures
Cite
@article{arxiv.2605.29987,
title = {MIC: Maximizing Informational Capacity in Adaptive Representations via Isotropic Subspace Alignment},
author = {Dang Hong Nguyen and Nhi Ngoc-Yen Nguyen and Huy-Hieu Pham},
journal= {arXiv preprint arXiv:2605.29987},
year = {2026}
}