Related papers: Entropic Regularization in the Deep Linear Network

Density Matrix Renormalization Group (DMRG) and its extensions in the form of Matrix Product States (MPS) are arguably the choice for the study of one dimensional quantum systems in the last three decades. However, due to the limited…

Dimensional regularization is a common method used to regulate the UV divergence of field theoretic quantities. When it is used in the context of Renyi entropy, however, it is important to consider whether such a procedure eliminates the…

Neural networks are playing a crucial role in everyday life, with the most modern generative models able to achieve impressive results. Nonetheless, their functioning is still not very clear, and several strategies have been adopted to…

We study regularized deep neural networks (DNNs) and introduce a convex analytic framework to characterize the structure of the hidden layers. We show that a set of optimal hidden layer weights for a norm regularized DNN training problem…

We summarize our renormalization group approach for the vector model as well as the matrix model which are the discretized quantum gravity in one- and two-dimensional spacetime. A difference equation is obtained which relates free energies…

Euclidean representations distort data with intrinsic non-Euclidean structure. While Riemannian representation learning offers a solution by embedding data onto matching manifolds, it typically relies on an encoder to estimate densities on…

Deep learning using multi-layer neural networks (NNs) architecture manifests superb power in modern machine learning systems. The trained Deep Neural Networks (DNNs) are typically large. The question we would like to address is whether it…

Layer normalization (LN) is a ubiquitous technique in deep learning but our theoretical understanding to it remains elusive. This paper investigates a new theoretical direction for LN, regarding to its nonlinearity and representation…

Riemannian neural networks, which extend deep learning techniques to Riemannian spaces, have gained significant attention in machine learning. To better classify the manifold-valued features, researchers have started extending Euclidean…

Quantum entanglement entropy has a geometric character. This is illustrated by the interpretation of Rindler space or black hole entropy as entanglement entropy. In general, one can define a "geometric entropy", associated with an event…

Diagonal linear networks (DLNs) are a toy simplification of artificial neural networks; they consist in a quadratic reparametrization of linear regression inducing a sparse implicit regularization. In this paper, we describe the trajectory…

A fully discrete Lagrangian scheme for solving a family of fourth order equations numerically is presented. The discretization is based on the equation's underlying gradient flow structure w.r.t. the $L^2$-Wasserstein distance, and adapts…

In this paper, we introduce a modular deep neural network (DNN) framework for data-driven reduced order modeling of dynamical systems relevant to fluid flows. We propose various deep neural network architectures which numerically predict…

We consider complexity of Deep Neural Networks (DNNs) and their associated massive over-parameterization. Such over-parametrization may entail susceptibility to adversarial attacks, loss of interpretability and adverse Size, Weight and…

We investigate the behavior of integral formulations of variable coefficient elliptic partial differential equations (PDEs) in the presence of steep internal layers. In one dimension, the equations that arise can be solved analytically and…

Deep neural networks (DNNs) have significantly advanced machine learning, with model depth playing a central role in their successes. The dynamical system modeling approach has recently emerged as a powerful framework, offering new…

Linear programming has played a crucial role in shaping decision-making, resource allocation, and cost reduction in various domains. In this paper, we investigate the application of overparametrized neural networks and their implicit bias…

Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to…

Implicit regularization is an important way to interpret neural networks. Recent theory starts to explain implicit regularization with the model of deep matrix factorization (DMF) and analyze the trajectory of discrete gradient dynamics in…

We revisit Deep Linear Discriminant Analysis (Deep LDA) from a likelihood-based perspective. While classical LDA is a simple Gaussian model with linear decision boundaries, attaching an LDA head to a neural encoder raises the question of…