Related papers: KKT-based optimality conditions for neural network…
Neural networks (NNs) have been extremely successful across many tasks in machine learning. Quantization of NN weights has become an important topic due to its impact on their energy efficiency, inference time and deployment on hardware.…
We identify necessary and sufficient conditions for a quantum channel to be optimal for any convex optimization problem in which the optimization is taken over the set of all quantum channels of a fixed size. Optimality conditions for…
We consider the convex optimization problem $\min \{f(x) : g_j(x)\leq 0, j=1,...,m\}$ where $f$ is convex, the feasible set K is convex and Slater's condition holds, but the functions $g_j$ are not necessarily convex. We show that for any…
We prove that finding all globally optimal two-layer ReLU neural networks can be performed by solving a convex optimization program with cone constraints. Our analysis is novel, characterizes all optimal solutions, and does not leverage…
The optimization problem behind neural networks is highly non-convex. Training with stochastic gradient descent and variants requires careful parameter tuning and provides no guarantee to achieve the global optimum. In contrast we show…
Linear classifiers and leaky ReLU networks trained by gradient flow on the logistic loss have an implicit bias towards solutions which satisfy the Karush--Kuhn--Tucker (KKT) conditions for margin maximization. In this work we establish a…
The paper proposes and justifies a new algorithm of the proximal Newton type to solve a broad class of nonsmooth composite convex optimization problems without strong convexity assumptions. Based on advanced notions and techniques of…
To every nearly convex optimization problem, that is a minimization problem with a nearly convex objective function and a nearly convex constraint set, we associate a uniquely defined convex optimization problem with a lower semicontinuous…
This paper presents how to perform minimax optimal classification, regression, and density estimation based on fixed-$k$ nearest neighbor (NN) searches. We consider a distributed learning scenario, in which a massive dataset is split into…
Training deep neural networks (DNNs) is an important and challenging optimization problem in machine learning due to its non-convexity and non-separable structure. The alternating minimization (AM) approaches split the composition structure…
This paper develops fundamental limits of deep neural network learning by characterizing what is possible if no constraints are imposed on the learning algorithm and on the amount of training data. Concretely, we consider Kolmogorov-optimal…
In this work, we investigate an indirect approach for the numerical solution of optimal control problems via neural networks. A customized neural network is constructed, where optimal state, co-state and control trajectories are…
The success of deep learning is due, to a large extent, to the remarkable effectiveness of gradient-based optimization methods applied to large neural networks. The purpose of this work is to propose a modern view and a general mathematical…
We propose a bilevel optimization strategy for selecting the best hyperparameter value for the nonsmooth $\ell_p$ regularizer with $0<p\le 1$. The concerned bilevel optimization problem has a nonsmooth, possibly nonconvex,…
Uniform polynomial approximation, also called minimax approximation or Chebyshev approximation, consists in searching polynomial approximation that minimizes the worst case error. Optimality conditions for the uniform approximation of…
This study introduces a novel approach to ensure the existence and uniqueness of optimal parameters in neural networks. The paper details how a recurrent neural networks (RNN) can be transformed into a contraction in a domain where its…
The paper introduces several new concepts for solving nonconvex or nonsmooth optimization problems, including convertible nonconvex function, exact convertible nonconvex function and differentiable convertible nonconvex function. It is…
In this paper, we examine an important problem of learning neural networks that certifiably meet certain specifications on input-output behaviors. Our strategy is to find an inner approximation of the set of admissible policy parameters,…
Sensor networks play a critical role in many situational awareness applications. In this paper, we study the problem of determining sensor placements to balance coverage and connectivity objectives over a target region. Leveraging algebraic…
The presence of non-convexity in smooth optimization problems arising from deep learning have sparked new smoothness conditions in the literature and corresponding convergence analyses. We discuss these smoothness conditions, order them,…