Related papers: A clustering heuristic to improve a derivative-fre…
Several machine learning applications involve the optimization of higher-order derivatives (e.g., gradients of gradients) during training, which can be expensive in respect to memory and computation even with automatic differentiation. As a…
In nonsmooth optimization, a negative subgradient is not necessarily a descent direction, making the design of convergent descent methods based on zeroth-order and first-order information a challenging task. The well-studied bundle methods…
In this paper, we propose a new method based on the Sliding Algorithm from Lan(2016, 2019) for the convex composite optimization problem that includes two terms: smooth one and non-smooth one. Our method uses the stochastic noised…
In this paper, we propose a generalized conditional gradient method for multiobjective optimization, which can be viewed as an improved extension of the classical Frank-Wolfe (conditional gradient) method for single-objective optimization.…
Hessian-free (HF) optimization has been successfully used for training deep autoencoders and recurrent networks. HF uses the conjugate gradient algorithm to construct update directions through curvature-vector products that can be computed…
Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are…
In this paper, we provide a novel analytical perspective on the theoretical understanding of gradient-based learning algorithms by interpreting consensus-based optimization (CBO), a recently proposed multi-particle derivative-free…
In this work, we consider methods for solving large-scale optimization problems with a possibly nonsmooth objective function. The key idea is to first specify a class of optimization algorithms using a generic iterative scheme involving…
We present a novel randomized block coordinate descent method for the minimization of a convex composite objective function. The method uses (approximate) partial second-order (curvature) information, so that the algorithm performance is…
A new depth-based clustering procedure for directional data is proposed. Such method is fully non-parametric and has the advantages to be flexible and applicable even in high dimensions when a suitable notion of depth is adopted. The…
In this paper, we consider mixed-integer nonsmooth constrained optimization problems whose objective/constraint functions are available only as the output of a black-box zeroth-order oracle (i.e., an oracle that does not provide derivative…
We propose a descent subgradient algorithm for unconstrained nonsmooth nonconvex multiobjective optimization problems. To find a descent direction, we present an iterative process that efficiently approximates the Goldstein subdifferential…
We propose a manifold sampling algorithm for minimizing a nonsmooth composition $f= h\circ F$, where we assume $h$ is nonsmooth and may be inexpensively computed in closed form and $F$ is smooth but its Jacobian may not be available. We…
In this paper, we present a heuristic adaptive fast gradient method. We show that in practice our method has a better convergence rate than popular today optimization methods. Moreover, we justify our method and point out some problems that…
We improve recently introduced consensus-based optimization method, proposed in [R. Pinnau, C. Totzeck, O. Tse and S. Martin, Math. Models Methods Appl. Sci., 27(01):183--204, 2017], which is a gradient-free optimization method for general…
This paper explores a method for solving constrained optimization problems when the derivatives of the objective function are unavailable, while the derivatives of the constraints are known. We allow the objective and constraint function to…
We consider minimization of a smooth nonconvex objective function using an iterative algorithm based on Newton's method and the linear conjugate gradient algorithm, with explicit detection and use of negative curvature directions for the…
In this paper, we will provide an introduction to the derivative-free optimization algorithms which can be potentially applied to train deep learning models. Existing deep learning model training is mostly based on the back propagation…
In this paper, we combine the positive aspects of the Gradient Sampling (GS) and bundle methods, as the most efficient methods in nonsmooth optimization, to develop a robust method for solving unconstrained nonsmooth convex optimization…
Optimization methods are essential in solving complex problems across various domains. In this research paper, we introduce a novel optimization method called Gaussian Crunching Search (GCS). Inspired by the behaviour of particles in a…