High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation
Abstract
State-of-the-art models rely on massive widths despite exhibiting low Intrinsic Dimension (ID). We posit that this redundancy serves the non-convex optimization search rather than the final representation. We validate this hypothesis by decoupling the solution geometry via data-independent random projections, demonstrating that ResNet, ViT, and BERT representations can be compressed by up to 16x with negligible performance degradation of around 1%. Notably, these oblivious projections achieve parity with PCA and learned baselines, confirming the solution manifold is intrinsically robust. These findings establish the foundation for Subspace-Native Distillation: a paradigm where student models target this intrinsic manifold directly, bypassing the high-dimensional optimization bottleneck to realize the vision of "Train Big, Deploy Small"
Cite
@article{arxiv.2512.23410,
title = {High-Dimensional Search, Low-Dimensional Solution: Decoupling Optimization from Representation},
author = {Yusuf Kalyoncuoglu and Ratmir Miftachov},
journal= {arXiv preprint arXiv:2512.23410},
year = {2026}
}
Comments
Code available at https://github.com/yuskal/Directly-Constructing-Low-Dimensional-Solution-Subspaces-in-Deep-Neural-Networks