Related papers: Understanding Mode Connectivity via Parameter Spac…
The critical locus of the loss function of a neural network is determined by the geometry of the functional space and by the parameterization of this space by the network's weights. We introduce a natural distinction between pure critical…
The loss surface of deep neural networks has recently attracted interest in the optimization and machine learning communities as a prime example of high-dimensional non-convex problem. Some insights were recently gained using spin glass…
In neuroscience, functional brain connectivity describes the connectivity between brain regions that share functional properties. Neuroscientists often characterize it by a time series of covariance matrices between functional measurements…
Reconstructing the states of the nodes of a dynamical network is a problem of fundamental importance in the study of neuronal and genetic networks. An underlying related problem is that of observability, i.e., identifying the conditions…
The vertex connectivity of a graph $G$ is the size of the smallest set of vertices $S$ such that $G \setminus S$ is disconnected. For the class of planar graphs, the problem of vertex connectivity is well-studied, both from structural and…
Characterization of real-world complex systems increasingly involves the study of their topological structure using graph theory. Among global network properties, small-world property, consisting in existence of relatively short paths…
Complex networks are the subject of fundamental interest from the scientific community at large. Several metrics have been introduced to characterize the structure of these networks, such as the degree distribution, degree correlation, path…
In this paper, we propose a geometric framework to analyze the convergence properties of gradient descent trajectories in the context of linear neural networks. We translate a well-known empirical observation of linear neural nets into a…
Current deep neural networks are highly overparameterized (up to billions of connection weights) and nonlinear. Yet they can fit data almost perfectly through variants of gradient descent algorithms and achieve unexpected levels of…
A quadratic approximation of neural network loss landscapes has been extensively used to study the optimization process of these networks. Though, it usually holds in a very small neighborhood of the minimum, it cannot explain many…
We consider the optimization problem associated with fitting two-layer ReLU networks with $k$ hidden neurons, where labels are assumed to be generated by a (teacher) neural network. We leverage the rich symmetry exhibited by such models to…
Symmetry -- invariance to certain operators -- is a fundamental concept in many branches of physics. We propose ways to measure symmetric properties of vertices, and their surroundings, in networks. To be stable to the randomness inherent…
High-throughput methods for yielding the set of connections in a neural system, the connectome, are now being developed. This tutorial describes ways to analyze the topological and spatial organization of the connectome at the macroscopic…
The energy landscape of high-dimensional non-convex optimization problems is crucial to understanding the effectiveness of modern deep neural network architectures. Recent works have experimentally shown that two different solutions found…
The idea of minimum curvilinearity (MC) is that the hidden geometry of complex networks, in particular when they are sufficiently sparse, clustered, small-world and heterogeneous, can be efficiently navigated using the minimum spanning tree…
Statistical neurodynamics studies macroscopic behaviors of randomly connected neural networks. We consider a deep layered feedforward network where input signals are processed layer by layer. The manifold of input signals is embedded in a…
In this paper, we study a generalization of the classical minimum cut prob- lem, called Connectivity Preserving Minimum Cut (CPMC) problem, which seeks a minimum cut to separate a pair (or pairs) of source and destination nodes and…
Recent research shows that sublevel sets of the loss surfaces of overparameterized networks are connected, exactly or approximately. We describe and compare experimentally a panel of methods used to connect two low-loss points by a low-loss…
Viewing neural network models in terms of their loss landscapes has a long history in the statistical mechanics approach to learning, and in recent years it has received attention within machine learning proper. Among other things, local…
We investigate the synchronization features of a network of spiking neurons under a distance-dependent coupling following a power-law model. The interplay between topology and coupling strength leads to the existence of different…