Related papers: On the Global Convergence of Particle Swarm Optimi…
Global minimization is a fundamental challenge in optimization, especially in machine learning, where finding the global minimum of a function directly impacts model performance and convergence. This article introduces a novel optimization…
All things in the world are interconnected, the only difference is the strength of their connections.Particle swarm optimization(PSO) simulates the foraging behavior of a flock of birds, information is transmitted to quickly find the…
Parameter updating is an important stage in parallelism-based distributed deep learning. Synchronous methods are widely used in distributed training the Deep Neural Networks (DNNs). To reduce the communication and synchronization overhead…
The search for the model or ingredients that describe the current vision of our cosmos has led to the creation of a set of highly favorable experiments, and therefore a great flow of information. Due to this torrent of information and the…
The purpose of this paper is to propose a time-step-robust cell-to-cell integration of particle trajectories in 3-D unstructured meshes in particle/mesh Lagrangian stochastic methods. The main idea is to dynamically update the mean fields…
Many tasks in machine learning and signal processing can be solved by minimizing a convex function of a measure. This includes sparse spikes deconvolution or training a neural network with a single hidden layer. For these problems, we study…
The Particle Swarm Optimization (PSO) algorithm is developed for solving the Schaffer F6 function in fewer than 4000 function evaluations on a total of 30 runs. Four variations of the Full Model of Particle Swarm Optimization (PSO)…
In recent years, several swarm intelligence optimization algorithms have been proposed to be applied for solving a variety of optimization problems. However, the values of several hyperparameters should be determined. For instance, although…
We introduce a new Swarm-Based Gradient Descent (SBGD) method for non-convex optimization. The swarm consists of agents, each is identified with a position, ${\mathbf x}$, and mass, $m$. The key to their dynamics is communication: masses…
Bi-level optimization problems, where one wishes to find the global minimizer of an upper-level objective function over the globally optimal solution set of a lower-level objective, arise in a variety of scenarios throughout science and…
We study stochastic optimization of nonconvex loss functions, which are typical objectives for training neural networks. We propose stochastic approximation algorithms which optimize a series of regularized, nonlinearized losses on large…
We introduce a novel first-order stochastic swarm intelligence (SI) model in the spirit of consensus formation models, namely a consensus-based optimization (CBO) algorithm, which may be used for the global optimization of a function in…
This paper proposes a stochastic gradient descent method with an adaptive Gaussian noise term for the global minimization of nearly convex functions, which are nonconvex and possess multiple strict local minimizers. The noise term,…
In this article, we propose a new approach, optimize then agree for minimizing a sum $ f = \sum_{i=1}^n f_i(x)$ of convex objective functions over a directed graph. The optimize then agree approach decouples the optimization step and the…
We propose novel particle swarm optimization (PSO) variants incorporated with deep neural networks (DNNs) for particles to pursue globally optimal positions in dynamic environments. PSO is a heuristic approach for solving complex…
In collective systems, the available agents are a limited resource that must be allocated among tasks to maximize collective performance. Computing the optimal allocation of several agents to numerous tasks through a brute-force approach…
In this work, we present a globalized stochastic semismooth Newton method for solving stochastic optimization problems involving smooth nonconvex and nonsmooth convex terms in the objective function. We assume that only noisy gradient and…
Solving statistical learning problems often involves nonconvex optimization. Despite the empirical success of nonconvex statistical optimization methods, their global dynamics, especially convergence to the desirable local minima, remain…
We discuss kinetic-based particle optimization methods and variable-sample strategies for problems where the cost function represents the expected value of a random mapping. Kinetic-based optimization methods rely on a consensus mechanism…
High-Dimensional and Incomplete matrices, which usually contain a large amount of valuable latent information, can be well represented by a Latent Factor Analysis model. The performance of an LFA model heavily rely on its optimization…