相关论文: Design of generalized fractional order gradient de…

Large-scale optimization problems require algorithms both effective and efficient. One such popular and proven algorithm is Stochastic Gradient Descent which uses first-order gradient information to solve these problems. This paper studies…

We study optimization algorithms for the finite sum problems frequently arising in machine learning applications. First, we propose novel variants of stochastic gradient descent with a variance reduction property that enables linear…

In this paper, we consider algorithms with integral action for solving online optimization problems characterized by quadratic cost functions with a time-varying optimal point described by an $(n-1)$th order polynomial. Using a version of…

Stochastic Gradient Descent (SGD) is the main approach to optimizing neural networks. Several generalization properties of deep networks, such as convergence to a flatter minima, are believed to arise from SGD. This article explores the…

Deep learning has aroused extensive attention due to its great empirical success. The efficiency of the block coordinate descent (BCD) methods has been recently demonstrated in deep neural network (DNN) training. However, theoretical…

In dual decomposition, the dual to an optimization problem with a specific structure is solved in distributed fashion using (sub)gradient and recently also fast gradient methods. The traditional dual decomposition suffers from two main…

In this paper, we consider finite difference approximations of the second order wave equation. We use finite difference operators satisfying the summation-by-parts property to discretize the equation in space. Boundary conditions and grid…

Stochastic Gradient Decent (SGD) is one of the core techniques behind the success of deep neural networks. The gradient provides information on the direction in which a function has the steepest rate of change. The main problem with basic…

Continual learning (CL) presents a fundamental challenge in training neural networks on sequential tasks without experiencing catastrophic forgetting. Traditionally, the dominant approach in CL has been gradient-based optimization, where…

In this paper, we establish new convergence results for the quantized distributed gradient descent and suggest a novel strategy of choosing the stepsizes for the high-performance of the algorithm. Under the strongly convexity assumption on…

This paper presents a way to define, classify and accelerate the order of convergence of an uncountable family of fractional fixed point methods, which may be useful to continue expanding the applications of fractional operators. The…

It is well known that using high-order numerical algorithms to solve fractional differential equations leads to almost the same computational cost with low-order ones but the accuracy (or convergence order) is greatly improved, due to the…

The paper studies numerical methods that preserve a Lyapunov function of a dynamical system, i.e. numerical approximations whose energy decreases, just like in the original differential equation. With this aim, a discrete gradient method is…

Radial Basis Function-generated Finite Differences (RBF-FD) is a meshless method that can be used to numerically solve partial differential equations. The solution procedure consists of two steps. First, the differential operator is…

The gradient descent (GD) method -- is a fundamental and likely the most popular optimization algorithm in machine learning (ML), with a history traced back to a paper in 1847 (Cauchy, 1847). It was studied under various assumptions,…

With the vigorous development of artificial intelligence technology, various engineering technology applications have been implemented one after another. The gradient descent method plays an important role in solving various optimization…

In the paper titled "New numerical approach for fractional differential equations" by A. Atangana and K.M. Owolabi [Math. Model. Nat. Phenom., 13(1), 2018], it is presented a method for the numerical solution of some fractional differential…

We present a novel randomized block coordinate descent method for the minimization of a convex composite objective function. The method uses (approximate) partial second-order (curvature) information, so that the algorithm performance is…

A scaled conjugate gradient method that accelerates existing adaptive methods utilizing stochastic gradients is proposed for solving nonconvex optimization problems with deep neural networks. It is shown theoretically that, whether with…

Computing the gradient of a function provides fundamental information about its behavior. This information is essential for several applications and algorithms across various fields. One common application that require gradients are…