Moreau envelope and proximal-point methods under the lens of high-order regularization

This paper is devoted to investigating the fundamental properties of the high-order proximal operator (HOPE) and the high-order Moreau envelope (HOME) in the nonconvex setting, where the quadratic regularization ($p=2$) is replaced by a $p$-order regularizer with $p > 1$. After establishing several basic properties of HOPE and HOME, we study the differentiability and weak smoothness of HOME under $q$-prox-regularity with $q \geq 2$ and $p$-calmness for $p \in (1,2]$ and $2 \leq p \leq q$. Furthermore, we propose a high-order proximal-point algorithm (HiPPA) and analyze the convergence of the generated sequence to proximal fixed points. Our results pave the way for the development of a high-order smoothing theory with $p>1$ that can lead to new algorithmic advances in the nonconvex setting. To illustrate this potential for nonsmooth and nonconvex optimization, we apply HiPPA to the Nesterov-Chebyshev-Rosenbrock functions.