Related papers: The star-shaped space of solutions of the spherica…
One of the most intriguing findings in the structure of neural network landscape is the phenomenon of mode connectivity: For two typical global minima, there exists a path connecting them without barrier. This concept of mode connectivity…
We study the binary and continuous negative-margin perceptrons as simple non-convex neural network models learning random rules and associations. We analyze the geometry of the landscape of solutions in both models and find important…
Neural computation in biological and artificial networks relies on the nonlinear summation of many inputs. The structural connectivity matrix of synaptic weights between neurons is a critical determinant of overall network function, but…
In these pedagogic notes I review the statistical mechanics approach to neural networks, focusing on the paradigmatic example of the perceptron architecture with binary an continuous weights, in the classification setting. I will review the…
It has recently been conjectured that neural network solution sets reachable via stochastic gradient descent (SGD) are convex, considering permutation invariances (Entezari et al., 2022). This means that a linear path can connect two…
The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a prime example of high-dimensional non-convex problem. Some insights were recently gained using spin glass…
Compositional energy-based models can generalize to larger combinatorial reasoning problems by reusing a learned factor energy across many local constraints. In our paper, we show that a key bottleneck in compositional reasoning is not…
The difficulty in exploring potential energy surfaces, which are nonconvex, stems from the presence of many local minima, typically separated by high barriers and often disconnected in configurational space. We obtain the global minimum on…
We study numerically the cluster structure of random ensembles of two NP-hard optimization problems originating in computational complexity, the vertex-cover problem and the number partitioning problem. We use branch-and-bound type…
We develop a statistical mechanical approach based on the replica method to study the design space of deep and wide neural networks constrained to meet a large number of training data. Specifically, we analyze the configuration space of the…
Network connectivity is usually addressed for convex domains where a direct line of sight exists between any two transmitting/receiving nodes. Here, we develop a general theory for the network connectivity properties across a small opening,…
The clear understanding of the non-convex landscape of neural network is a complex incomplete problem. This paper studies the landscape of linear (residual) network, the simplified version of the nonlinear network. By treating the gradient…
Statistical-physics calculations in machine learning and theoretical neuroscience often involve lengthy derivations that obscure physical interpretation. Here, we give concise, non-replica derivations of several key results and highlight…
The functional features of spatial networks depend upon a non-trivial relationship between the topological and physical structure. Here, we explore that relationship for spatial networks with radial symmetry and disordered fractal…
In this paper, we investigate the geometric structure of activation spaces of fully connected layers in neural networks and then show applications of this study. We propose an efficient approximation algorithm to characterize the convex…
Energy-minimizing constraint maps are a natural extension of the obstacle problem within a vectorial framework. Due to inherent topological constraints, these maps manifest a diverse structure that includes singularities similar to harmonic…
The training of neural networks is a complex, high-dimensional, non-convex and noisy optimization problem whose theoretical understanding is interesting both from an applicative perspective and for fundamental reasons. A core challenge is…
The study of random landscapes has long relied on counting stationary points: metastable states and the barriers between them. However, this method is useless for describing flat regions, common in constraint satisfaction problems. We…
We propose a dynamical neural network model with a hierarchical and modular structure. The network architecture can be derived by minimizing an energy function that is originally designed based on two kinds of neurons with quite different…
Distributing points on a (possibly high-dimensional) sphere with minimal energy is a long-standing problem in and outside the field of mathematics. This paper considers a novel energy function that arises naturally from statistics and…