Related papers: Higher-order Reverse Automatic Differentiation wit…
It is well known that finding a global optimum is extremely challenging for nonconvex optimization. There are some recent efforts \cite{anandkumar2016efficient, cartis2018second, cartis2020sharp, chen2019high} regarding the optimization…
This paper is concerned with the directional derivative of the value function for a very general set-constrained optimization problem under perturbation. Under reasonable assumptions, we obtain upper and lower estimates for the upper and…
Gaussian processes with derivative information are useful in many settings where derivative information is available, including numerous Bayesian optimization and regression tasks that arise in the natural sciences. Incorporating derivative…
We introduce a high-order numerical scheme for fractional ordinary differential equations with the Caputo derivative. The method is developed by dividing the domain into a number of subintervals, and applying the quadratic interpolation on…
The object of the present paper is to extend the third-order iterative method for solving nonlinear equations into systems of nonlinear equations. Since our motive is to develop the method which improve the order of convergence of Newton's…
A new method to represent and approximate rotation matrices is introduced. The method represents approximations of a rotation matrix $Q$ with linearithmic complexity, i.e. with $\frac{1}{2}n\lg(n)$ rotations over pairs of coordinates,…
Zeroth-order optimization is an important research topic in machine learning. In recent years, it has become a key tool in black-box adversarial attack to neural network based image classifiers. However, existing zeroth-order optimization…
We present a simple formalism for the calculation of the derivatives of the electronic density matrix at any order, within density functional theory. Our approach, contrary to previous ones, is not based on the perturbative expansion of the…
We established a new eighth-order iterative method, consisting of three steps, for solving nonlinear equations. Per iteration the method requires four evaluations (three function evaluations and one evaluation of the first derivative).…
In this paper, we propose a second order optimization method to learn models where both the dimensionality of the parameter space and the number of training samples is high. In our method, we construct on each iteration a Krylov subspace…
We propose a high order numerical homogenization method for dissipative ordinary differential equations (ODEs) containing two time scales. Essentially, only first order homogenized model globally in time can be derived. To achieve a high…
In a Hilbert space setting, for convex optimization, we show the convergence of the iterates to optimal solutions for a class of accelerated first-order algorithms. They can be interpreted as discrete temporal versions of an inertial…
We develop an exact coordinate descent algorithm for high-dimensional regularized Huber regression. In contrast to composite gradient descent methods, our algorithm fully exploits the advantages of coordinate descent when the underlying…
This letter presents a method to reduce the computational demands of including second-order dynamics sensitivity information into the Differential Dynamic Programming (DDP) trajectory optimization algorithm. An approach to DDP is developed…
This work investigates finite differences and the use of interpolation models to obtain approximations to the first and second derivatives of a function. Here, it is shown that if a particular set of points is used in the interpolation…
We present a new accelerated stochastic second-order method that is robust to both gradient and Hessian inexactness, which occurs typically in machine learning. We establish theoretical lower bounds and prove that our algorithm achieves…
This work presents a novel matrix-based method for constructing an approximation Hessian using only function evaluations. The method requires less computational power than interpolation-based methods and is easy to implement in matrix-based…
In this paper, we consider an unconstrained optimization model where the objective is a sum of a large number of possibly nonconvex functions, though overall the objective is assumed to be smooth and convex. Our bid to solving such model…
Numerical methods for fractional calculus attract increasing interests due to its wide applications in various fields such as physics, mechanics, etc. In this paper, we focus on constructing high-order algorithms for Riesz derivatives,…
The depth of networks plays a crucial role in the effectiveness of deep learning. However, the memory requirement for backpropagation scales linearly with the number of layers, which leads to memory bottlenecks during training. Moreover,…