Related papers: Relativized depth
Overparameterized fully-connected neural networks have been shown to behave like kernel models when trained with gradient descent, under mild conditions on the width, the learning rate, and the parameter initialization. In the limit of…
While the universal approximation property holds both for hierarchical and shallow networks, we prove that deep (hierarchical) networks can approximate the class of compositional functions with the same accuracy as shallow networks but with…
Large language models have demonstrated remarkable capabilities across many tasks, yet face significant challenges when dealing with recursive reasoning problems, those requiring the resolution of nested hierarchical structures. While prior…
In this paper we introduce a new formulation of Bennett's logical depth based on pebble transducers. This notion is defined based on the difference between the minimal length descriptional complexity of prefixes of infinite sequences from…
Kernelization is the standard framework to analyze preprocessing routines mathematically. Here, in terms of efficiency, we demand the preprocessing routine to run in time polynomial in the input size. However, today, various NP-complete…
A triplet comparison oracle on a set $S$ takes an object $x \in S$ and for any pair $\{y, z\} \subset S \setminus \{x\}$ declares which of $y$ and $z$ is more similar to $x$. Partitioned Local Depth (PaLD) supplies a principled…
People believe that depth plays an important role in success of deep neural networks (DNN). However, this belief lacks solid theoretical justifications as far as we know. We investigate role of depth from perspective of margin bound. In…
In this work we study the mutual benefits of two common computer vision tasks, self-supervised depth estimation and semantic segmentation from images. For example, to help unsupervised monocular depth estimation, constraints from semantic…
Data depth is a concept in multivariate statistics that measures the centrality of a point in a given data cloud in $\IR^d$. If the depth of a point can be represented as the minimum of the depths with respect to all one-dimensional…
Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the implicit…
Many constructions in computability theory rely on "time tricks". In the higher setting, relativising to some oracles shows the necessity of these. We construct an oracle~$A$ and a set~$X$, higher Turing reducible to~$X$, but for which…
The statistical rank tests play important roles in univariate non-parametric data analysis. If one attempts to generalize the rank tests to a multivariate case, the problem of defining a multivariate order will occur. It is not clear how to…
We prove various results connected together by the common thread of computability theory. First, we investigate a new notion of algorithmic dimension, the inescapable dimension, which lies between the effective Hausdorff and packing…
The unwavering success of deep learning in the past decade led to the increasing prevalence of deep learning methods in various application fields. However, the downsides of deep learning, most prominently its lack of trustworthiness, may…
Understanding the relationship between the depth of a neural network and its representational capacity is a central problem in deep learning theory. In this work, we develop a geometric framework to analyze the expressivity of ReLU networks…
Why and when does depth improve generalization? We study this question in an implementation-agnostic state-transition model, where a depth-$k$ predictor is a readout class $H$ composed with the word ball $B(k,F)$ generated by hidden state…
We present three different perspectives of oracle. First, an oracle is a blackbox; second, an oracle is a tool to change the way we access mathematical objects; and third, an oracle is a factor that causes a change in truth values.…
A relatively new topic in computability theory is the study of notions of computation that are robust against mistakes on some kind of small set. However, despite the recent popularity of this topic relatively foundational questions about…
Enclosing depth is a recently introduced depth measure which gives a lower bound to many depth measures studied in the literature. So far, enclosing depth has only been studied from a combinatorial perspective. In this work, we give the…
Not only are Deep Neural Networks (DNNs) black box models, but also we frequently conceptualize them as such. We lack good interpretations of the mechanisms linking inputs to outputs. Therefore, we find it difficult to analyze in…