Related papers: Depth, Highness and DNR degrees

Deep neural networks with skip-connections, such as ResNet, show excellent performance in various image classification benchmarks. It is though observed that the initial motivation behind them - training deeper networks - does not actually…

Let $T_n$ be a random recursive tree with $n$ nodes. List vertices of $T_n$ in decreasing order of degree as $v^1,\ldots,v^n$, and write $d^i$ and $h^i$ for the degree of $v^i$ and the distance of $v^i$ from the root, respectively. We prove…

Deep neural networks are highly expressive models that have recently achieved state of the art performance on speech and visual recognition tasks. While their expressiveness is the reason they succeed, it also causes them to learn…

Deep Neural Networks (DNN) have been widely used to carry out segmentation tasks in both electron and light microscopy. Most DNNs developed for this purpose are based on some variation of the encoder-decoder type U-Net architecture, in…

We introduce a methodology for designing and training deep neural networks (DNN) that we call "Deep Regression Ensembles" (DRE). It bridges the gap between DNN and two-layer neural networks trained with random feature regression. Each layer…

Feature spaces in the deep layers of convolutional neural networks (CNNs) are often very high-dimensional and difficult to interpret. However, convolutional layers consist of multiple channels that are activated by different types of…

The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and the impact of initialization. One key…

Unsupervised pretraining and dropout have been well studied, especially with respect to regularization and output consistency. However, our understanding about the explicit convergence rates of the parameter estimates, and their dependence…

In sentence modeling and classification, convolutional neural network approaches have recently achieved state-of-the-art results, but all such efforts process word vectors sequentially and neglect long-distance dependencies. To exploit both…

Existing analyses of optimization in deep learning are either continuous, focusing on (variants of) gradient flow, or discrete, directly treating (variants of) gradient descent. Gradient flow is amenable to theoretical analysis, but is…

We propose a deep learning-based solution for the problem of feature learning in one-class classification. The proposed method operates on top of a Convolutional Neural Network (CNN) of choice and produces descriptive features while…

One of the mysteries in the success of neural networks is randomly initialized first order methods like gradient descent can achieve zero training loss even though the objective function is non-convex and non-smooth. This paper demystifies…

The choice of learning rate (LR) functions and policies has evolved from a simple fixed LR to the decaying LR and the cyclic LR, aiming to improve the accuracy and reduce the training time of Deep Neural Networks (DNNs). This paper presents…

Recently, deep architectures, such as recurrent and recursive neural networks have been successfully applied to various natural language processing tasks. Inspired by bidirectional recurrent neural networks which use representations that…

Very deep convolutional neural networks (CNNs) yield state of the art results on a wide variety of visual recognition problems. A number of state of the the art methods for image recognition are based on networks with well over 100 layers…

Feature learning is thought to be one of the fundamental reasons for the success of deep neural networks. It is rigorously known that in two-layer fully-connected neural networks under certain conditions, one step of gradient descent on the…

In a companion paper it was shown that the class of constant-depth determinate k-ary recursive clauses is efficiently learnable. In this paper we present negative results showing that any natural generalization of this class is hard to…

Complex-valued neural networks (CVNNs) have recently been successful in various pioneering areas which involve wave-typed information and frequency-domain processing. This work addresses different structures and classification of CVNNs. The…

Let alpha = a_1 a_2 ... a_n be a sequence of nonnegative integers. The ascent set of alpha, Asc(alpha), consists of all indices k where a_{k+1} > a_k. An ascent sequence is alpha where the growth of the a_k is bounded by the elements of…

Constant-recursive sequences are those which satisfy a linear recurrence, so that later terms can be obtained as a linear combination of the previous ones. The rank of a constant-recursive sequence is the minimal number of previous terms…