A Distributional View of High Dimensional Optimization
Abstract
This PhD thesis presents a distributional view of optimization in place of a worst-case perspective. We motivate this view with an investigation of the failure point of classical optimization. Subsequently we consider the optimization of a randomly drawn objective function. This is the setting of Bayesian Optimization. After a review of Bayesian optimization we outline how such a distributional view may explain predictable progress of optimization in high dimension. It further turns out that this distributional view provides insights into optimal step size control of gradient descent. To enable these results, we develop mathematical tools to deal with random input to random functions and a characterization of non-stationary isotropic covariance kernels. Finally, we outline how assumptions about the data, specifically exchangability, can lead to random objective functions in machine learning and analyze their landscape.
Cite
@article{arxiv.2507.16315,
title = {A Distributional View of High Dimensional Optimization},
author = {Felix Benning},
journal= {arXiv preprint arXiv:2507.16315},
year = {2025}
}
Comments
Most chapters reproduces work that was conducted during my PhD. The review of classical worst-case optimization and Bayesian Optimization is unpublished and may present a novel perspective. While it is not difficult to do, building Machine Learning Theory from exchangeable data is also fairly non-standard and offers an intuitive explanation for many canonical loss functions