Related papers: Geometric Approaches to Matrix Normalization and G…
Contraction analysis establishes exponential incremental convergence of a nonlinear system by solving a linear matrix inequality for a contraction metric, and has become a standard resource for solving problems in nonlinear control and…
Low-rank matrix estimation plays a central role in various applications across science and engineering. Recently, nonconvex formulations based on matrix factorization are provably solved by simple gradient descent algorithms with strong…
This paper considers the analysis of continuous time gradient-based optimization algorithms through the lens of nonlinear contraction theory. It demonstrates that in the case of a time-invariant objective, most elementary results on…
We study the implicit regularization imposed by gradient descent for learning multi-layer homogeneous functions including feed-forward fully connected and convolutional deep neural networks with linear, ReLU or Leaky ReLU activation. We…
Originating from spectral graph theory, cospectrality is a powerful generalization of exchange symmetry and can be applied to all real-valued symmetric matrices. Two vertices of an undirected graph with real edge weights are cospectral iff…
Although it is relatively easy to apply, the gradient method often displays a disappointingly slow rate of convergence. Its convergence is specially based on the structure of the matrix of the algebraic linear system, and on the choice of…
In this paper, we derive nearly tight probabilistic norm bounds for a class of random matrices we call graph matrices. While the classical case of symmetric matrices with independent random entries (Wigner's matrices) is a special case, in…
A systematic mismatch exists between mathematically ideal and effective activation updates during gradient descent. As intended, parameters update in their direction of steepest descent. However, activations are argued to constitute a more…
In this article, we propose a new type of square matrix associated with an undirected graph by trading off the naturally imbedded symmetry in them. The proposed matrix is defined using the neighbourhood sets of the vertices. It is called as…
We investigate connections between the symmetries (automorphisms) of a graph and its spectral properties. Whenever a graph has a symmetry, i.e. a nontrivial automorphism $\phi$, it is possible to use $\phi$ to decompose any matrix…
We consider the nonconvex minimization problem, with quartic objective function, that arises in the exact recovery of a configuration matrix $P\in \R^{nd}$ of $n$ points when a Euclidean distance matrix, \EDMp, is given with embedding…
This paper establishes risk convergence and asymptotic weight matrix alignment --- a form of implicit regularization --- of gradient flow and gradient descent when applied to deep linear networks on linearly separable data. In more detail,…
Recent studies have shown that many nonconvex machine learning problems satisfy a generalized-smooth condition that extends beyond traditional smooth nonconvex optimization. However, the existing algorithms are not fully adapted to such…
Natural gradients have been widely studied from both theoretical and empirical perspectives, and it is commonly believed that natural gradients have advantages over standard (Euclidean) gradients in capturing the intrinsic geometric…
Problems related to graph matching and isomorphisms are very important both from a theoretical and practical perspective, with applications ranging from image and video analysis to biological and biomedical problems. The graph matching…
We investigate matrix models in three dimensions where the global $\text{SU}(N)$ symmetry acts via the adjoint map. Analyzing their ground state which is homogeneous in space and can carry either a unique or multiple fixed charges, we show…
Active contour models based on partial differential equations have proved successful in image segmentation, yet the study of their geometric formulation on arbitrary geometric graphs is still at an early stage. In this paper, we introduce…
This paper develops expansive gradient dynamics in deep neural network-induced mapping spaces. Specifically, we generate tools and concepts for minimizing a class of energy functionals in an abstract Hilbert space setting covering a wide…
As science and engineering have become increasingly data-driven, the role of optimization has expanded to touch almost every stage of the data analysis pipeline, from signal and data acquisition to modeling and prediction. The optimization…
In this paper, we present some theoretical work to explain why simple gradient descent methods are so successful in solving non-convex optimization problems in learning large-scale neural networks (NN). After introducing a mathematical tool…