Related papers: Faster Convergence for Transformer Fine-tuning wit…

We propose a descent subgradient algorithm for unconstrained nonsmooth nonconvex multiobjective optimization problems. To find a descent direction, we present an iterative process that efficiently approximates the Goldstein subdifferential…

The concept of learning to optimize involves utilizing a trainable optimization strategy rather than relying on manually defined full gradient estimations such as ADAM. We present a framework that jointly trains the full gradient estimator…

We study decentralized optimization over networks where agents cooperatively minimize a smooth (strongly) convex sum of local losses while communicating only with immediate neighbors. Prevailing decentralized methods require either…

We propose a novel linesearch variant of the trust region normal map-based semismooth Newton method developed in [Ouyang and Milzarek, Math. Program. 212(1-2), 389--435 (2025)] for solving a class of nonsmooth, nonconvex composite-type…

We propose a first-order method for solving inequality constrained optimization problems. The method is derived from our previous work [12], a modified search direction method (MSDM) that applies the singular-value decomposition of…

A new pattern search method for bound constrained optimization is introduced. The proposed algorithm employs the coordinate directions, in a suitable way, with a nonmonotone line search for accepting the new iterate, without using…

This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular…

Adaptive gradient-based optimizers such as Adagrad and Adam are crucial for achieving state-of-the-art performance in machine translation and language modeling. However, these methods maintain second-order statistics for each parameter,…

Consensus maximization is widely used for robust fitting in computer vision. However, solving it exactly, i.e., finding the globally optimal solution, is intractable. A* tree search, which has been shown to be fixed-parameter tractable, is…

In this paper, we consider augmented Lagrangian (AL) algorithms for solving large-scale nonlinear optimization problems that execute adaptive strategies for updating the penalty parameter. Our work is motivated by the recently proposed…

The adaptive moment estimation (Adam) optimizer proposed by Kingma & Ba (2014) is presumably the most popular stochastic gradient descent (SGD) optimization method for the training of deep neural networks (DNNs) in artificial intelligence…

LLM alignment remains a critical challenge. Inference-time methods provide a flexible alternative to fine-tuning, but their uniform computational effort often yields suboptimal alignment. We hypothesize that for many alignment tasks, the…

Understanding the remarkable efficacy of Adam when training transformer-based language models has become a central research topic within the optimization community. To gain deeper insights, several simplifications of Adam have been…

A line search in gradient-based optimization algorithm solves the problem of determining the optimal learning rate for a given gradient or search direction in a single iteration. For most problems, this is determined by evaluating different…

Transducer models have emerged as a promising choice for end-to-end ASR systems, offering a balanced trade-off between recognition accuracy, streaming capabilities, and inference speed in greedy decoding. However, beam search significantly…

Projected-search methods for bound-constrained optimization are based on performing a search along a piecewise-linear continuous path obtained by projecting a search direction onto the feasible region. A benefit of these methods is that…

The classical line search for learning rate (LR) tuning in the stochastic gradient descent (SGD) algorithm can tame the convergence slowdown due to data-sampling noise. In a federated setting, wherein the client heterogeneity introduces a…

We present a rigorous convergence analysis of a new method for density-based topology optimization that provides point-wise bound preserving design updates and faster convergence than other popular first-order topology optimization methods.…

Adaptive first-order optimizers are fundamental tools in deep learning, although they may suffer from poor generalization due to the nonuniform gradient scaling. In this work, we propose AdamL, a novel variant of the Adam optimizer, that…

Adam-type methods, the extension of adaptive gradient methods, have shown great performance in the training of both supervised and unsupervised machine learning models. In particular, Adam-type optimizers have been widely used empirically…