Related papers: Function-Space Optimality of Neural Architectures …

This work performs a study of the category of complete matrix-normed spaces, called matricial Banach spaces. Many of the usual constructions of Banach spaces extend in a natural way to matricial Banach spaces, including products, direct…

Deep neural networks typically treat nonlinearities as fixed primitives (e.g., ReLU), limiting both interpretability and the granularity of control over the induced function class. While recent additive models (like KANs) attempt to address…

The efficacy of deep neural networks is heavily reliant on the design of non-linear activation functions, yet existing approaches often struggle to balance optimization stability with computational efficiency. While piecewise linear…

While it is well known that nonlinear methods of approximation can often perform dramatically better than linear methods, there are still questions on how to measure the optimal performance possible for such methods. This paper studies…

Focusing on establishing a mathematical basis for kernel methods in sparse multi-task learning, we explore the theory of vector-valued reproducing kernel Banach spaces (RKBSs) endowed with $\ell_{p,1}$-norms ($1\le p\le +\infty$),…

Motivated by a seminal paper of professor M. Z. Nashed published in 1987 on classification of ill-posed linear operator equations and distinguishing two types of ill-posedness in Banach and Hilbert spaces, we present, illustrate and justify…

Recurrent Neural Networks (RNNs) were recently successfully used to model the way neural activity drives task-related behavior in animals, operating under the implicit assumption that the obtained solutions are universal. Observations in…

Recent research has suggested that the brain is more shallow than previously thought, challenging the traditionally assumed hierarchical structure of the ventral visual pathway. Here, we demonstrate that optimizing convolutional network…

We study lower bounds for the norm of the product of polynomials and their applications to the so called \emph{plank problem.} We are particularly interested in polynomials on finite dimensional Banach spaces, in which case our results…

Neural networks are usually not the tool of choice for nonparametric high-dimensional problems where the number of input features is much larger than the number of observations. Though neural networks can approximate complex multivariate…

Path regularization has shown to be a very effective regularization to train neural networks, leading to a better generalization property than common regularizations i.e. weight decay, etc. We propose a first near-complete (as will be made…

The neural network is a powerful computing framework that has been exploited by biological evolution and by humans for solving diverse problems. Although the computational capabilities of neural networks are determined by their structure,…

In these notes we study a family of Banach spaces, denoted $\, D^{0,\,\al}(\Ov)\,,$ $\,\al \in\,\R^+\,,$ and called H-log spaces. For $\,0<\,\la\leq\,1\,,$ one has $ C^{0,\,\la}(\Ov)\subset D^{0,\,\al}(\Ov) \subset\,C(\Ov)\,,$ with compact…

Weight sharing has become a de facto standard in neural architecture search because it enables the search to be done on commodity hardware. However, recent works have empirically shown a ranking disorder between the performance of…

Sparsity of a learning solution is a desirable feature in machine learning. Certain reproducing kernel Banach spaces (RKBSs) are appropriate hypothesis spaces for sparse learning methods. The goal of this paper is to understand what kind of…

Neural networks have shown tremendous potential for reconstructing high-resolution images in inverse problems. The non-convex and opaque nature of neural networks, however, hinders their utility in sensitive applications such as medical…

Neural architecture search (NAS) enables finding the best-performing architecture from a search space automatically. Most NAS methods exploit an over-parameterized network (i.e., a supernet) containing all possible architectures (i.e.,…

In the present work, we discuss variational regularization for ill-posed nonlinear problems with focus on an oversmoothing penalty term. This means in our model that the searched-for solution of the considered nonlinear operator equation…

Network Utility Maximization (NUM) is often applied for the cross-layer design of wireless networks considering known wireless channels. However, realistic wireless channel capacities are stochastic bearing time-varying statistics,…

In this paper, we study total variation (TV)-regularized training of infinite-width shallow ReLU neural networks, formulated as a convex optimization problem over measures on the unit sphere. Our approach leverages the duality theory of…