A consensus-based global optimization method for high dimensional machine learning problems
Abstract
We improve recently introduced consensus-based optimization method, proposed in [R. Pinnau, C. Totzeck, O. Tse and S. Martin, Math. Models Methods Appl. Sci., 27(01):183--204, 2017], which is a gradient-free optimization method for general non-convex functions. We first replace the isotropic geometric Brownian motion by the component-wise one, thus removing the dimensionality dependence of the drift rate, making the method more competitive for high dimensional optimization problems. Secondly, we utilize the random mini-batch ideas to reduce the computational cost of calculating the weighted average which the individual particles tend to relax toward. For its mean-field limit--a nonlinear Fokker-Planck equation--we prove, in both time continuous and semi-discrete settings, that the convergence of the method, which is exponential in time, is guaranteed with parameter constraints {\it independent} of the dimensionality. We also conduct numerical tests to high dimensional problems to check the success rate of the method.
Cite
@article{arxiv.1909.09249,
title = {A consensus-based global optimization method for high dimensional machine learning problems},
author = {José A. Carrillo and Shi Jin and Lei Li and Yuhua Zhu},
journal= {arXiv preprint arXiv:1909.09249},
year = {2020}
}