Related papers: An Augmented Lagrangian Method for Training Recurr…

This paper presents a fully asynchronous and distributed approach for tackling optimization problems in which both the objective function and the constraints may be nonconvex. In the considered network setting each node is active upon…

Advances in optical and electrophysiological recording technologies have made it possible to record the dynamics of thousands of neurons, opening up new possibilities for interpreting and controlling large neural populations in behaving…

Symmetric cone programming covers a broad class of convex optimization problems, including linear programming, second-order cone programming, and semidefinite programming. Although the augmented Lagrangian method (ALM) is well-suited for…

We consider regression problems with binary weights. Such optimization problems are ubiquitous in quantized learning models and digital communication systems. A natural approach is to optimize the corresponding Lagrangian using variants of…

Recurrent neural networks (RNN) have been successfully applied to various sequential decision-making tasks, natural language processing applications, and time-series predictions. Such networks are usually trained through back-propagation…

Training a one-node neural network with ReLU activation function (One-Node-ReLU) is a fundamental optimization problem in deep learning. In this paper, we begin with proving the NP-hardness of training One-Node-ReLU. We then present an…

An efficient algorithm for recurrent neural network training is presented. The approach increases the training speed for tasks where a length of the input sequence may vary significantly. The proposed approach is based on the optimal batch…

We propose a variant of the classical augmented Lagrangian method for constrained optimization problems in Banach spaces. Our theoretical framework does not require any convexity or second-order assumptions and allows the treatment of…

Recurrent neural networks (RNNs) serve as a fundamental building block for many sequence tasks across natural language processing. Recent research has focused on recurrent dropout techniques or custom RNN cells in order to improve…

In this paper, we consider nonconvex optimization problems with nonsmooth nonconvex objective function and nonlinear equality constraints. We assume that both the objective function and the functional constraints can be separated into 2…

Augmented Lagrangian method (ALM) has been popularly used for solving constrained optimization problems. Practically, subproblems for updating primal variables in the framework of ALM usually can only be solved inexactly. The convergence…

Gradient descent (GD) type optimization schemes are the standard methods to train artificial neural networks (ANNs) with rectified linear unit (ReLU) activation. Such schemes can be considered as discretizations of gradient flows (GFs)…

We propose a data-driven technique to automatically learn contextual uncertainty sets in robust optimization, resulting in excellent worst-case and average-case performance while also guaranteeing constraint satisfaction. Our method…

In the past years, augmented Lagrangian methods have been successfully applied to several classes of non-convex optimization problems, inspiring new developments in both theory and practice. In this paper we bring most of these recent…

A common problem in training neural networks is the vanishing and/or exploding gradient problem which is more prominently seen in training of Recurrent Neural Networks (RNNs). Thus several algorithms have been proposed for training RNNs.…

We present a numerical method for the minimization of constrained optimization problems where the objective is augmented with large quadratic penalties of inconsistent equality constraints. Such objectives arise from quadratic integral…

The Augmented Lagrangian Method (ALM) is an iterative method for the solution of equality-constrained non-linear programming problems. In contrast to the quadratic penalty method, the ALM can satisfy equality constraints in an exact way.…

We present an architecture of a recurrent neural network (RNN) with a fully-connected deep neural network (DNN) as its feature extractor. The RNN is equipped with both causal temporal prediction and non-causal look-ahead, via…

Recurrent Neural Networks (RNNs) and their variants, such as Long-Short Term Memory (LSTM) networks, and Gated Recurrent Unit (GRU) networks, have achieved promising performance in sequential data modeling. The hidden layers in RNNs can be…

Zero-One Composite Optimization (0/1-COP) is a prototype of nonsmooth, nonconvex optimization problems and it has attracted much attention recently. The augmented Lagrangian Method (ALM) has stood out as a leading methodology for such…