Related papers: FUSE: First-Order and Second-Order Unified SynthEs…

Federated learning (FL) has emerged as a widely adopted training paradigm for privacy-preserving machine learning. While the SGD-based FL algorithms have demonstrated considerable success in the past, there is a growing trend towards…

This paper is dedicated to the construction of high-order (in both space and time) finite-difference schemes for both forward and backward PDEs and PIDEs, such that option prices obtained by solving both the forward and backward equations…

Stochastic bilevel optimization generalizes the classic stochastic optimization from the minimization of a single objective to the minimization of an objective function that depends the solution of another optimization problem. Recently,…

First order methods, which solely rely on gradient information, are commonly used in diverse machine learning (ML) and data analysis (DA) applications. This is attributed to the simplicity of their implementations, as well as low…

In this paper, we introduce various mechanisms to obtain accelerated first-order stochastic optimization algorithms when the objective function is convex or strongly convex. Specifically, we extend the Catalyst approach originally designed…

We study the optimization of non-convex functions that are not necessarily smooth (gradient and/or Hessian are Lipschitz) using first order methods. Smoothness is a restrictive assumption in machine learning in both theory and practice,…

Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a…

We present a new high-order accurate discretisation on unstructured meshes of quadrilateral elements. Our Face Upwinded Spectral Element (FUSE) method uses the same node distribution as a high-order continuous Galerkin (CG) method, but with…

We introduce AdaSub, a stochastic optimization algorithm that computes a search direction based on second-order information in a low-dimensional subspace that is defined adaptively based on available current and past information. Compared…

In this paper, we discuss distributed optimization over directed graphs, where doubly-stochastic weights cannot be constructed. Most of the existing algorithms overcome this issue by applying push-sum consensus, which utilizes…

In this paper, we study pooling downstream beds across specialties in a stochastic operating room planning problem. The main sources of uncertainty are stochastic surgical durations and patients' lengths of stay. We developed a two-stage…

Efficient data selection is crucial for enhancing the training efficiency of deep neural networks and minimizing annotation requirements. Traditional methods often face high computational costs, limiting their scalability and practical use.…

Despite an extensive body of literature on deep learning optimization, our current understanding of what makes an optimization algorithm effective is fragmented. In particular, we do not understand well whether enhanced optimization…

Product formula methods, particularly the second-order Suzuki decomposition, are an important tool for simulating quantum dynamics on quantum computers due to their simplicity and unitarity preservation. While higher-order schemes have been…

In a wide range of applications it is desirable to optimally control a dynamical system with respect to concurrent, potentially competing goals. This gives rise to a multiobjective optimal control problem where, instead of computing a…

The trade-off among accuracy, robustness, and computational cost remains a key challenge in simulating complex flows. Second-order schemes are computationally efficient but lack the accuracy required for resolving intricate flow structures,…

Particle swam optimization (PSO) is a popular stochastic optimization method that has found wide applications in diverse fields. However, PSO suffers from high computational complexity and slow convergence speed. High computational…

In this paper, we study smooth stochastic multi-level composition optimization problems, where the objective function is a nested composition of $T$ functions. We assume access to noisy evaluations of the functions and their gradients,…

Shuffling strategies for stochastic gradient descent (SGD), including incremental gradient, shuffle-once, and random reshuffling, are supported by rigorous convergence analyses for arbitrary within-epoch permutations. In particular, random…

We propose a First-Order System Least Squares (FOSLS) method based on deep-learning for numerically solving second-order elliptic PDEs. The method we propose is capable of dealing with either variational and non-variational problems, and…