Related papers: Depth, Highness and DNR degrees

Bennett's notion of depth is usually considered to describe the usefulness and internal organization of the information encoded into an object such as an infinite binary sequence. We consider a natural way to relativize the notion of depth…

In this article, we study the relationship between notions of depth for sequences, namely, Bennett's notions of strong and weak depth, and deep $\Pi^0_1$ classes, introduced by the authors and motivated by previous work of Levin. For the…

Richter, Stephan, and Zhang asked whether every nonrecursive many-one degree contains a least finite-one degree. We prove this for every nonrecursive \ce\ many-one degree containing a $D$-maximal set. The proof handles the simple cases via…

We investigate a variety of statistical properties associated with the number of distinct degrees that exist in a typical network for various classes of networks. For a single realization of a network with N nodes that is drawn from an…

Richter, Stephan, and Zhang asked whether every nonrecursive many-one degree contains a least finite-one degree. We solve this question in the negative, already within the class of computably enumerable many-one degrees. Positive answers…

An infinite binary sequence is Bennett deep if, for any computable time bound, the difference between the time-bounded prefix-free Kolmogorov complexity and the prefix-free Kolmogorov complexity of its initial segments is eventually…

A remarkable achievement in algorithmic randomness and algorithmic information theory was the discovery of the notions of K-trivial, K-low and Martin-Lof-random-low sets: three different definitions turns out to be equivalent for very…

We construct an increasing $\omega$-sequence $(a_n)$ of Turing degrees which forms an initial segment of the Turing degrees, and such that each~$a_{n+1}$ is diagonally noncomputable relative to $a_n$. It follows that the~$\mathsf{DNR}$…

In this note we consider the $k$th level of the uniform random recursive tree after $n$ steps, and prove that the proportion of nodes with degree greater than $t\log n$ converges to $(1-t)^k$ almost surely, as $n\to\infty$, for every…

For any positive integer $k$, there exist neural networks with $\Theta(k^3)$ layers, $\Theta(1)$ nodes per layer, and $\Theta(1)$ distinct parameters which can not be approximated by networks with $\mathcal{O}(k)$ layers unless they are…

Any gradient descent optimization requires to choose a learning rate. With deeper and deeper models, tuning that learning rate can easily become tedious and does not necessarily lead to an ideal convergence. We propose a variation of the…

We study neural network training (NNT): optimizing a neural network's parameters to minimize the training loss over a given dataset. NNT has been studied extensively under theoretic lenses, mainly on two-layer networks with linear or ReLU…

We study the dynamics of gradient descent on objective functions of the form $f(\prod_{i=1}^{k} w_i)$ (with respect to scalar parameters $w_1,\ldots,w_k$), which arise in the context of training depth-$k$ linear neural networks. We prove…

Convolutional Neural Networks (CNNs) has revolutionized computer vision, but training very deep networks has been challenging due to the vanishing gradient problem. This paper explores Residual Networks (ResNet), introduced by He et al.…

Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has recently emerged as a powerful framework, offering new…

In this manuscript, we study the learning of deep attention neural networks, defined as the composition of multiple self-attention layers, with tied and low-rank weights. We first establish a mapping of such models to sequence multi-index…

Let $(x_n)_{n\geq0}$ be a linear recurrence of order $k\geq2$ satisfying $$x_n=a_1x_{n-1}+a_2x_{n-2}+\dots+a_kx_{n-k}$$ for all integers $n\geq k$, where $a_1,\dots,a_k,x_0,\dots, x_{k-1}\in \mathbb{Z},$ with $a_k\neq0$. In [`The quotient…

A candidate explanation of the good empirical performance of deep neural networks is the implicit regularization effect of first order optimization methods. Inspired by this, we prove a convergence theorem for nonconvex composite…

In this thesis, we develop various techniques for working with sets in machine learning. Each input or output is not an image or a sequence, but a set: an unordered collection of multiple objects, each object described by a feature vector.…

We construct a set of strong recurrence which is not a van der Corput set. This shows that the class of enhanced van der Corput sets is a proper subclass of sets of strong recurrence. In addition, we derive that the class of sets of strong…