Related papers: A Recovery Guarantee for Sparse Neural Networks
Previous work has demonstrated that MLPs within ReLU Transformers exhibit high levels of sparsity, with many of their activations equal to zero for any given token. We build on that work to more deeply explore how token-level sparsity…
Decision trees are widely used for their low computational cost, good predictive performance, and ability to assess the importance of features. Though often used in practice for feature selection, the theoretical guarantees of these methods…
Overparameterized neural networks can interpolate a given dataset in many different ways, prompting the fundamental question: which among these solutions should we prefer, and what explicit regularization strategies will provably yield…
The Strong Lottery Ticket Hypothesis (SLTH) posits that large, randomly initialized neural networks contain sparse subnetworks capable of approximating a target function at initialization without training, suggesting that pruning alone is…
We study the problem of jointly sparse support recovery with 1-bit compressive measurements in a sensor network. Sensors are assumed to observe sparse signals having the same but unknown sparse support. Each sensor quantizes its measurement…
In this paper, we introduce a weighted $\ell_2/\ell_1$ minimization to recover block sparse signals with arbitrary prior support information. When partial prior support information is available, a sufficient condition based on the high…
This paper introduces recovery thresholding hyperinterpolations, a novel class of methods for sparse signal reconstruction in the presence of noise. We develop a framework that integrates thresholding operators--including hard thresholding,…
We consider a structured estimation problem where an observed matrix is assumed to be generated as an $s$-sparse linear combination of $N$ given $n\times n$ positive-semidefinite matrices. Recovering the unknown $N$-dimensional and…
Intensively growing approach in signal processing and acquisition, the Compressive Sensing approach, allows sparse signals to be recovered from small number of randomly acquired signal coefficients. This paper analyses some of the commonly…
Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks -- those with more connections than the size of the training data -- are often able to memorize the training data with $100\%$ accuracy. This…
We present a novel stagewise strategy for improving greedy algorithms for sparse recovery. We demonstrate its efficiency both for synthesis and analysis sparse priors, where in both cases we demonstrate its computational efficiency and…
Neural networks can be trained to solve regression problems by using gradient-based methods to minimize the square loss. However, practitioners often prefer to reformulate regression as a classification problem, observing that training on…
Deep neural networks have emerged as powerful tools for learning operators defined over infinite-dimensional function spaces. However, existing theories frequently encounter difficulties related to dimensionality and limited…
Deep neural networks tend to underestimate uncertainty and produce overly confident predictions. Recently proposed solutions, such as MC Dropout and SDENet, require complex training and/or auxiliary out-of-distribution data. We propose a…
The observation that activation sparsity emerges in MLP blocks of standardly trained Transformers offers an opportunity to drastically reduce computation costs without sacrificing performance. To theoretically explain this phenomenon,…
We consider the problem of sparse signal recovery from noisy measurements. Many of frequently used recovery methods rely on some sort of tuning depending on either noise or signal parameters. If no estimates for either of them are…
The possibility for one to recover the parameters-weights and biases-of a neural network thanks to the knowledge of its function on a subset of the input space can be, depending on the situation, a curse or a blessing. On one hand,…
In this paper, we consider the sparse phase retrieval problem, recovering an $s$-sparse signal $\bm{x}^{\natural}\in\mathbb{R}^n$ from $m$ phaseless samples $y_i=|\langle\bm{x}^{\natural},\bm{a}_i\rangle|$ for $i=1,\ldots,m$. Existing…
Plasticity Loss is an increasingly important phenomenon that refers to the empirical observation that as a neural network is continually trained on a sequence of changing tasks, its ability to adapt to a new task diminishes over time. We…
Recent advances in artificial intelligence have relied heavily on increasingly large neural networks, raising concerns about their computational and environmental costs. This paper investigates whether simpler, sparser networks can maintain…