Related papers: Introducing memory to a family of multi-step multi…

In several recently proposed stochastic optimization methods (e.g. RMSProp, Adam, Adadelta), parameter updates are scaled by the inverse square roots of exponential moving averages of squared past gradients. Maintaining these per-parameter…

We consider the problem of unconstrained minimization of a smooth objective function in $\mathbb{R}^d$ in setting where only function evaluations are possible. We propose and analyze stochastic zeroth-order method with heavy ball momentum.…

Value iteration is a fixed point iteration technique utilized to obtain the optimal value function and policy in a discounted reward Markov Decision Process (MDP). Here, a contraction operator is constructed and applied repeatedly to arrive…

In this note we consider splitting methods based on linear multistep methods and stabilizing corrections. To enhance the stability of the methods, we employ an idea of Bruno & Cubillos (2016) who combine a high-order extrapolation formula…

We investigate the convergence properties of a stochastic primal-dual splitting algorithm for solving structured monotone inclusions involving the sum of a cocoercive operator and a composite monotone operator. The proposed method is the…

We propose a novel time stepping method for linear poroelasticity by extending a recent iterative decoupling approach to the second-order case. This results in a two-step scheme with an inner iteration and a relaxation step. We prove…

It is well known that the numerical solution of the Non-Fickian flows at the current stage depends on all previous time instances. Consequently, the storage requirement increases linearly, while the computational complexity grows…

A development of an inverse first-order divided difference operator for functions of several variables is presented. Two generalized derivative-free algorithms builded up from Ostrowski's method for solving systems of nonlinear equations…

We derive unconditionally stable and convergent variable-step BDF2 scheme for solving the MBE model with slope selection. The discrete orthogonal convolution kernels of the variable-step BDF2 method is commonly utilized recently for solving…

Mathematical reasoning has been challenging for large language models (LLMs), and the introduction of step-by-step Chain-of-Thought (CoT) inference has significantly advanced the mathematical capabilities of LLMs. However, current…

In this paper, we propose a novel accelerated stochastic gradient method with momentum, which momentum is the weighted average of previous gradients. The weights decays inverse proportionally with the iteration times. Stochastic gradient…

Feature selection is a widely used dimension reduction technique to select feature subsets because of its interpretability. Many methods have been proposed and achieved good results, in which the relationships between adjacent data points…

Deep model merging represents an emerging research direction that combines multiple fine-tuned models to harness their specialized capabilities across different tasks and domains. Current model merging techniques focus on merging all…

This paper proposes several explicit and implicit multistep frequency response optimized integrators considering first or second order derivative. A prediction-based method aiming at accelerating a novel power system transient simulation…

The $\gamma$-FDP and $k$-FWER multiple testing error metrics, which are tail probabilities of the respective error statistics, have become popular recently as less-stringent alternatives to the FDR and FWER. We propose general and flexible…

This work considers stepsize schedules for gradient descent on smooth convex objectives. We extend the existing literature and propose a unified technique for constructing stepsizes with analytic bounds for an arbitrary number of…

Gradient descent is slow to converge for ill-conditioned problems and non-convex problems. An important technique for acceleration is step-size adaptation. The first part of this paper contains a detailed review of step-size adaptation…

Multitask learning assumes that models capable of learning from multiple tasks can achieve better quality and efficiency via knowledge transfer, a key feature of human learning. Though, state of the art ML models rely on high customization…

We suggest a simple adaptive step-size procedure, which does not require any line-search, for a general class of nonlinear optimization methods and prove convergence of a general method under mild assumptions. In particular, the goal…

We develop an estimation methodology for a factor model for high-dimensional matrix-valued time series, where common stochastic trends and common stationary factors can be present. We study, in particular, the estimation of (row and column)…