相关论文: Weight Norm Control

In this work, we consider an optimal control problem subject to a nonlinear PDE constraint and apply it to the regularized $p$-Laplace equation. To this end, a reduced unconstrained optimization problem in terms of the control variable is…

This work proposes a fully decentralized strategy for maintaining the formation rigidity of a multi-robot system using only range measurements, while still allowing the graph topology to change freely over time. In this direction, a first…

Prior support constrained compressed sensing, achieved via the weighted norm minimization, has of late become popular due to its potential for applications. For the weighted norm minimization problem, $$ min \|x\|_{p,w} \text{ subject to }…

Neural Collapse (NC) refers to the emergence of highly symmetric geometric structures in the representations of deep neural networks during the terminal phase of training. Despite its prevalence, the theoretical understanding of NC remains…

Deep neural networks are learning models with a very high capacity and therefore prone to over-fitting. Many regularization techniques such as Dropout, DropConnect, and weight decay all attempt to solve the problem of over-fitting by…

Normalization techniques are a boon for modern deep learning. They let weights converge more quickly with often better generalization performances. It has been argued that the normalization-induced scale invariance among the weights…

This work is substituted by the paper in arXiv:2011.14066. Stochastic gradient descent is the de facto algorithm for training deep neural networks (DNNs). Despite its popularity, it still requires fine tuning in order to achieve its best…

In this paper we study the structure of solutions of the one dimensional weighted total variation regularisation problem, motivated by its application in signal recovery tasks. We study in depth the relationship between the weight function…

In this paper we discuss the numerical solution of elliptic distributed optimal control problems with state or control constraints when the control is considered in the energy norm. As in the unconstrained case we can relate the…

With the success of deep neural networks (NNs) in a variety of domains, the computational and storage requirements for training and deploying large NNs have become a bottleneck for further improvements. Sparsification has consequently…

This paper studies the convergence of the mirror descent algorithm for finite horizon stochastic control problems with measure-valued control processes. The control objective involves a convex regularisation function, denoted as $h$, with…

With the development of deep neural networks, the size of network models becomes larger and larger. Model compression has become an urgent need for deploying these network models to mobile or embedded devices. Model quantization is a…

In the real open world, data tends to follow long-tailed class distributions, motivating the well-studied long-tailed recognition (LTR) problem. Naive training produces models that are biased toward common classes in terms of higher…

Following Demidovich's concept and definition of convergent systems, we analyze the optimal nonlinear damping control, recently proposed [1] for the second-order systems. Targeting the problem of output regulation, correspondingly tracking…

While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized…

A unified solution to adaptive approximation-based control for nonlinear systems with accurate and inaccurate state measurement is synthesized in this study. Starting from the standard adaptive approximation-based controller with accurate…

We study the matrix completion problem when the observation pattern is deterministic and possibly non-uniform. We propose a simple and efficient debiased projection scheme for recovery from noisy observations and analyze the error under a…

Adaptive learning rate methods have been successfully applied in many fields, especially in training deep neural networks. Recent results have shown that adaptive methods with exponential increasing weights on squared past gradients (i.e.,…

This paper studies the problem of mapping optimization in decentralized control problems. A global optimization algorithm is proposed based on the ideas of ``deterministic annealing" - a powerful non-convex optimization framework derived…

We present Amos, a stochastic gradient-based optimizer designed for training deep neural networks. It can be viewed as an Adam optimizer with theoretically supported, adaptive learning-rate decay and weight decay. A key insight behind Amos…