English

Coordinate-wise Armijo's condition: General case

Optimization and Control 2020-03-12 v1 Machine Learning Dynamical Systems Machine Learning

Abstract

Let z=(x,y)z=(x,y) be coordinates for the product space Rm1×Rm2\mathbb{R}^{m_1}\times \mathbb{R}^{m_2}. Let f:Rm1×Rm2Rf:\mathbb{R}^{m_1}\times \mathbb{R}^{m_2}\rightarrow \mathbb{R} be a C1C^1 function, and f=(xf,yf)\nabla f=(\partial _xf,\partial _yf) its gradient. Fix 0<α<10<\alpha <1. For a point (x,y)Rm1×Rm2(x,y) \in \mathbb{R}^{m_1}\times \mathbb{R}^{m_2}, a number δ>0\delta >0 satisfies Armijo's condition at (x,y)(x,y) if the following inequality holds: \begin{eqnarray*} f(x-\delta \partial _xf,y-\delta \partial _yf)-f(x,y)\leq -\alpha \delta (||\partial _xf||^2+||\partial _yf||^2). \end{eqnarray*} In one previous paper, we proposed the following {\bf coordinate-wise} Armijo's condition. Fix again 0<α<10<\alpha <1. A pair of positive numbers δ1,δ2>0\delta _1,\delta _2>0 satisfies the coordinate-wise variant of Armijo's condition at (x,y)(x,y) if the following inequality holds: \begin{eqnarray*} [f(x-\delta _1\partial _xf(x,y), y-\delta _2\partial _y f(x,y))]-[f(x,y)]\leq -\alpha (\delta _1||\partial _xf(x,y)||^2+\delta _2||\partial _yf(x,y)||^2). \end{eqnarray*} Previously we applied this condition for functions of the form f(x,y)=f(x)+g(y)f(x,y)=f(x)+g(y), and proved various convergent results for them. For a general function, it is crucial - for being able to do real computations - to have a systematic algorithm for obtaining δ1\delta _1 and δ2\delta _2 satisfying the coordinate-wise version of Armijo's condition, much like Backtracking for the usual Armijo's condition. In this paper we propose such an algorithm, and prove according convergent results. We then analyse and present experimental results for some functions such as f(x,y)=ax+yf(x,y)=a|x|+y (given by Asl and Overton in connection to Wolfe's method), f(x,y)=x3sin(1/x)+y3sin(1/y)f(x,y)=x^3 sin (1/x) + y^3 sin(1/y) and Rosenbrock's function.

Cite

@article{arxiv.2003.05252,
  title  = {Coordinate-wise Armijo's condition: General case},
  author = {Tuyen Trung Truong},
  journal= {arXiv preprint arXiv:2003.05252},
  year   = {2020}
}

Comments

6 pages. Preprint arXiv:1911.07820 is incorporated as a very special case

R2 v1 2026-06-23T14:11:30.535Z