相关论文: Geometric Regularization from Overparameterization

Double descent presents a counter-intuitive aspect within the machine learning domain, and researchers have observed its manifestation in various models and tasks. While some theoretical explanations have been proposed for this phenomenon…

Classical regression has a simple geometric description in terms of a projection of the training labels onto the column space of the design matrix. However, for over-parameterized models -- where the number of fit parameters is large enough…

We present new exact solutions for two-dimensional geometries generated by continuous distributions of topological defects within a conformal metric framework. By reformulating Einstein's equations in two dimensions as a Poisson equation…

A major challenge in understanding the generalization of deep learning is to explain why (stochastic) gradient descent can exploit the network architecture to find solutions that have good generalization performance when using high capacity…

We study the high-dimensional training dynamics of a shallow neural network with quadratic activation in a teacher-student setup. We focus on the extensive-width regime, where the teacher and student network widths scale proportionally with…

Many modern neural network architectures are trained in an overparameterized regime where the parameters of the model exceed the size of the training dataset. Sufficiently overparameterized neural network architectures in principle have the…

This work introduces topological regularization as a framework for handling ultraviolet divergences in quantum field theory, reinterpreting infinities as topological obstructions at spacetime boundaries. Through geometric compactification…

Neural networks have shown tremendous potential for reconstructing high-resolution images in inverse problems. The non-convex and opaque nature of neural networks, however, hinders their utility in sensitive applications such as medical…

Geometric embeddings have recently received attention for their natural ability to represent transitive asymmetric relations via containment. Box embeddings, where objects are represented by n-dimensional hyperrectangles, are a particularly…

Common practice in modern machine learning involves fitting a large number of parameters relative to the number of observations. These overparameterized models can exhibit surprising generalization behavior, e.g., ``double descent'' in the…

We describe the equivalence at one loop between constrained differential renormalization and regularization by dimensional reduction in the MS scheme. To illustrate it, we reexamine the calculation of supergravity corrections to (g-2)_l.

Grokking -- the abrupt transition from memorization to generalization long after near-zero training loss -- has been studied mainly in single-task settings. We extend geometric analysis to multi-task modular arithmetic, training…

Regularization is a core component of modern inverse problems, as it helps establish the well-posedness of the solution of interest. Popular regularization approaches include variational regularization and iterative regularization. The…

The generalization mystery in deep learning is the following: Why do over-parameterized neural networks trained with gradient descent (GD) generalize well on real datasets even though they are capable of fitting random datasets of…

While there are many studies on weight regularization, the study on structure regularization is rare. Many existing systems on structured prediction focus on increasing the level of structural dependencies within the model. However, this…

The isoperimetric problem with a density or weighting seeks to enclose prescribed weighted volume with minimum weighted perimeter. According to Chambers' recent proof of the log-convex density conjecture, for many densities on…

The unprecedented prowess of measurement techniques provides a detailed, multi-scale look into the depths of living systems. Understanding these avalanches of high-dimensional data -- by distilling underlying principles and mechanisms --…

Training materials through periodic drive allows to endow materials and structures with complex elastic functions. As a result of the driving, the system explores the high dimensional space of structures, ultimately converging to a…

Recently, we developed a theory of a geometrically growing system. Here we show that the theory can explain some phenomena of power-law distribution including classical demographic and economic and novel pandemic instances, without…

A regularization procedure, that allows one to relate singularities of curvature to those of the Einstein tensor without some of the shortcomings of previous approaches, is proposed. This regularization is obtained by requiring that (i) the…