Nonmonotone higher-order Taylor approximation methods for composite problems

We study composite optimization problems in which the smooth part of the objective function is $p$-times continuously differentiable, where $p \geq 1$ is an integer. Higher-order methods are known to be effective for solving such problems, as they speed up convergence rates. These methods often require, or implicitly ensure, a monotonic decrease in the objective function across iterations. Maintaining this monotonicity typically requires that the $p$-th derivative of the smooth part of the objective function is globally Lipschitz or that the generated iterates remain bounded. In this paper, we propose nonmonotone higher-order Taylor approximation (NHOTA) method for composite problems. Our method achieves the same nice global and rate of convergence properties as traditional higher-order methods while eliminating the need for global Lipschitz continuity assumptions, strict descent condition, or explicit boundedness of the iterates. Specifically, for nonconvex composite problems, we derive global convergence rate to a stationary point of order $\mathcal{O}(k^{-\frac{p}{p+1}})$, where $k$ is the iteration counter. Moreover, when the objective function satisfies the Kurdyka-{\L}ojasiewicz (KL) property, we obtain improved rates that depend on the KL parameter. Furthermore, for convex composite problems, our method achieves sublinear convergence rate of order $\mathcal{O}(k^{-p})$ in function values. Finally, preliminary numerical experiments on nonconvex phase retrieval problems highlight the promising performance of the proposed approach.