Related papers: ReLU Code Space: A Basis for Rating Network Qualit…
Feed-forward networks can be interpreted as mappings with linear decision surfaces at the level of the last layer. We investigate how the tangent space of the network can be exploited to refine the decision in case of ReLU (Rectified Linear…
It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset. As a specific case, we consider estimating the…
We provide a theoretical algorithm for checking local optimality and escaping saddles at nondifferentiable points of empirical risks of two-layer ReLU networks. Our algorithm receives any parameter value and returns: local minimum,…
This paper provides novel insights into channel and subspace codes in nonadaptive channel sensing with a single RF chain. Observing that this problem naturally maps to a noncoherent decoding problem, we show that the sensing performance of…
We present empirical evidence that neural networks with ReLU and Absolute Value activations learn distance-based representations. We independently manipulate both distance and intensity properties of internal activations in trained models,…
The parameter space for any fixed architecture of feedforward ReLU neural networks serves as a proxy during training for the associated class of functions - but how faithful is this representation? It is known that many different parameter…
Neural Networks (NNs) are the method of choice for building learning algorithms. Their popularity stems from their empirical success on several challenging learning problems. However, most scholars agree that a convincing theoretical…
The input space of a neural network with ReLU-like activations is partitioned into multiple linear regions, each corresponding to a specific activation pattern of the included ReLU-like activations. We demonstrate that this partition…
Independent parallel q-ary symmetric channels are a suitable transmission model for several applications. The proposed weighted-Hamming metric is tailored to this setting and enables optimal decoding performance. We show that some…
Overwhelming theoretical and empirical evidence shows that mildly overparametrized neural networks -- those with more connections than the size of the training data -- are often able to memorize the training data with $100\%$ accuracy. This…
While neural networks have achieved high performance in different learning tasks, their accuracy drops significantly in the presence of small adversarial perturbations to inputs. Defenses based on regularization and adversarial training are…
The rank metric measures the distance between two matrices by the rank of their difference. Codes designed for the rank metric have attracted considerable attention in recent years, reinforced by network coding and further motivated by a…
Sum-rank Hamming codes are introduced in this work. They are essentially defined as the longest codes (thus of highest information rate) with minimum sum-rank distance at least $ 3 $ (thus one-error-correcting) for a fixed redundancy $ r $,…
Hashing based cross-modal retrieval has recently made significant progress. But straightforward embedding data from different modalities into a joint Hamming space will inevitably produce false codes due to the intrinsic modality…
Rank modulation is a way of encoding information to correct errors in flash memory devices as well as impulse noise in transmission lines. Modeling rank modulation involves construction of packings of the space of permutations equipped with…
We consider the stability analysis of feedback systems with rectified linear unit (ReLU) activations, and model this problem with polynomial optimization. Stability can be certified by means of copositive multipliers in the framework of…
Error control is significant to network coding, since when unchecked, errors greatly deteriorate the throughput gains of network coding and seriously undermine both reliability and security of data. Two families of codes, subspace and rank…
Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization. In this work, we show…
We present the theory of linear rank-metric codes from the point of view of their fundamental parameters. These are: the minimum rank distance, the rank distribution, the maximum rank, the covering radius, and the field size. The focus of…
In this letter we propose a new hybrid code called "RM-Polar" codes. This new codes are constructed by combining the construction of Reed-Muller (RM) code and Polar code. It has much larger minimum Hamming distance than Polar codes,…