Related papers: Entropic Regularization in the Deep Linear Network
In arXiv:1609.03493, the authors extended the exact renormalization group (ERG) to arbitrary wave-functionals in quantum field theory (QFT). Applying this formalism, we show that the ERG flow of density matrices is given by a Lindblad…
Deep neural networks (DNNs) have achieved significant success in a variety of real world applications, i.e., image classification. However, tons of parameters in the networks restrict the efficiency of neural networks due to the large model…
We derive an inequality for the linear entropy, that gives sharp bounds for all finite dimensional systems. The derivation is based on generalised Bloch decompositions and provides a strict improvement for the possible distribution of…
Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem…
The recently discovered Neural collapse (NC) phenomenon states that the last-layer weights of Deep Neural Networks (DNN), converge to the so-called Equiangular Tight Frame (ETF) simplex, at the terminal phase of their training. This ETF…
Dropout Regularization, serving to reduce variance, is nearly ubiquitous in Deep Learning models. We explore the relationship between the dropout rate and model complexity by training 2,000 neural networks configured with random…
The generalization error of deep neural networks via their classification margin is studied in this work. Our approach is based on the Jacobian matrix of a deep neural network and can be applied to networks with arbitrary non-linearities…
We develop a density matrix renormalization group (DMRG) algorithm for constrained quantum lattice models that successfully {\it{implements the local constraints as symmetries in the contraction of the matrix product states and matrix…
We consider the LP in standard form min {c T x\,: Ax = b; x $\ge$ 0} and inspired by $\epsilon$-regularization in Optimal Transport, we introduce its $\epsilon$-regularization ''min {c T x + $\epsilon$ f (x)\,: Ax = b; x $\ge$ 0}'' via the…
We study the conjectured relationship between the implicit regularization in neural networks, trained with gradient-based methods, and rank minimization of their weight matrices. Previously, it was proved that for linear networks (of depth…
Deep learning utilizing deep neural networks (DNNs) has achieved a lot of success recently in many important areas such as computer vision, natural language processing, and recommendation systems. The lack of convexity for DNNs has been…
In neural networks, developing regularization algorithms to settle overfitting is one of the major study areas. We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new…
In this paper, we explicitly determine local and global minimizers of the $\mathcal{L}^2$ cost function in underparametrized Deep Learning (DL) networks; our main goal is to shed light on their geometric structure and properties. We…
The coefficient of the logarithmic term in the entropy on even spheres is re-computed by the local technique of integrating the finite temperature energy density up to the horizon on static d--dimensional de Sitter space and thence finding…
Low-dimensional structure in real-world data plays an important role in the success of generative models, which motivates diffusion models defined on intrinsic data manifolds. Such models are driven by stochastic differential equations…
While deep learning is successful in a number of applications, it is not yet well understood theoretically. A satisfactory theoretical characterization of deep learning however, is beginning to emerge. It covers the following questions: 1)…
The manipulator workspace mapping is an important problem in robotics and has attracted significant attention in the community. However, most of the pre-existing algorithms have expensive time complexity due to the reliance on sophisticated…
We study the role of $L_2$ regularization in deep learning, and uncover simple relations between the performance of the model, the $L_2$ coefficient, the learning rate, and the number of training steps. These empirical relations hold when…
The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a prime example of high-dimensional non-convex problem. Some insights were recently gained using spin glass…
Products of truncated unitary matrices, independently and uniformly drawn from the unitary group, can be used to study universal aspects of monitored quantum circuits. The von Neumann entropy of the corresponding density matrix decreases…