Related papers: Lockout: Sparse Regularization of Neural Networks
Conventional algorithms for sparse signal recovery and sparse representation rely on $l_1$-norm regularized variational methods. However, when applied to the reconstruction of $\textit{sparse images}$, i.e., images where only a few pixels…
$L_p$-norm regularization schemes such as $L_0$, $L_1$, and $L_2$-norm regularization and $L_p$-norm-based regularization techniques such as weight decay, LASSO, and elastic net compute a quantity which depends on model weights considered…
In this work, we investigate the use of sparsity-inducing regularizers during training of Convolution Neural Networks (CNNs). These regularizers encourage that fewer connections in the convolution and fully connected layers take non-zero…
Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy. Recently, techniques based on dynamic reallocation of…
Federated Learning (FL) has attracted much interest due to the significant advantages it brings to training deep neural network (DNN) models. However, since communications and computation resources are limited, training DNN models in FL…
Beside the minimization of the prediction error, two of the most desirable properties of a regression scheme are stability and interpretability. Driven by these principles, we propose continuous-domain formulations for one-dimensional…
Pruning methods have recently grown in popularity as an effective way to reduce the size and computational complexity of deep neural networks. Large numbers of parameters can be removed from trained models with little discernible loss in…
We propose Sparse Neural Network architectures that are based on random or structured bipartite graph topologies. Sparse architectures provide compression of the models learned and speed-ups of computations, they can also surpass their…
Sparse recovery is widely applied in many fields, since many signals or vectors can be sparsely represented under some frames or dictionaries. Most of fast algorithms at present are based on solving $l^0$ or $l^1$ minimization problems and…
This article aims to provide a comprehensive overview of sparse optimization, with a focus on both sparse signal recovery and sparse regularization techniques. We will begin by exploring the foundations of sparse optimization, delving into…
Recently established equivalences between differential equations and the structure of neural networks enabled some interpretation of training of a neural network as partial-differential-equation (PDE) constrained optimization. We add to the…
The robustness and anomaly detection capability of neural networks are crucial topics for their safe adoption in the real-world. Moreover, the over-parameterization of recent networks comes with high computational costs and raises questions…
The compression of deep neural networks (DNNs) to reduce inference cost becomes increasingly important to meet realistic deployment requirements of various applications. There have been a significant amount of work regarding network…
We study two factors in neural network training: data parallelism and sparsity; here, data parallelism means processing training data in parallel using distributed systems (or equivalently increasing batch size), so that training can be…
As a tractable approach, regularization is frequently adopted in sparse optimization. This gives rise to the regularized optimization, aiming at minimizing the $\ell_0$ norm or its continuous surrogates that characterize the sparsity. From…
Regularization-based approaches for injecting constraints in Machine Learning (ML) were introduced to improve a predictive model via expert knowledge. We tackle the issue of finding the right balance between the loss (the accuracy of the…
We consider the ill-posed inverse problem of identifying a nonlinearity in a time-dependent PDE model. The nonlinearity is approximated by a neural network, and needs to be determined alongside other unknown physical parameters and the…
We study a regularization framework that combines a convex fidelity term with multiple $\ell_1$-based regularizers, each linked to a distinct linear transform. This multi-penalty model enhances flexibility in promoting structured sparsity.…
Recent work on deep neural network pruning has shown there exist sparse subnetworks that achieve equal or improved accuracy, training time, and loss using fewer network parameters when compared to their dense counterparts. Orthogonal to…
Sparse neural networks have been widely applied to reduce the computational demands of training and deploying over-parameterized deep neural networks. For inference acceleration, methods that discover a sparse network from a pre-trained…