Related papers: Understanding Gradient Descent through the Trainin…

We study the deformation of the input space by a trained autoencoder via the Jacobians of the trained weight matrices. In doing so, we prove bounds for the mean squared errors for points in the input space, under assumptions regarding the…

Deep learning involves a difficult non-convex optimization problem with a large number of weights between any two adjacent layers of a deep structure. To handle large data sets or complicated networks, distributed training is needed, but…

Recent experiments have shown that training trajectories of multiple deep neural networks with different architectures, optimization algorithms, hyper-parameter settings, and regularization methods evolve on a remarkably low-dimensional…

We study the gradients of a maxout network with respect to inputs and parameters and obtain bounds for the moments depending on the architecture and the parameter distribution. We observe that the distribution of the input-output Jacobian…

Gradient descent during the learning process of a neural network can be subject to many instabilities. The spectral density of the Jacobian is a key component for analyzing stability. Following the works of Pennington et al., such Jacobians…

We introduce a weak notion of $2\times 2$-minors of gradients of a suitable subclass of $BV$ functions. In the case of maps in $BV(\mathbb{R}^2;\mathbb{R}^2)$ such a notion extends the standard definition of Jacobian determinant to…

In the realm of fractal geometry, intricate structures emerge from simple iterative processes that partition parameter spaces into regions of stability and instability. Likewise, training large language models involves iteratively applying…

Robustness to perturbation is a key topic in the study of complex systems occurring across a wide variety of applications from epidemiology to biochemistry. Here we analyze the eigenspectrum of the Jacobian matrices associated to a general…

This paper investigates low-rank structure in the gradients of the training loss for two-layer neural networks while relaxing the usual isotropy assumptions on the training data and parameters. We consider a spiked data model in which the…

This work introduces the concept of tangent space regularization for neural-network models of dynamical systems. The tangent space to the dynamics function of many physical systems of interest in control applications exhibits useful…

Neural networks have been successfully used for classification tasks in a rapidly growing number of practical applications. Despite their popularity and widespread use, there are still many aspects of training and classification that are…

Genes are connected in regulatory networks, often modelled by ordinary differential equations. Changes in expression of a gene propagate to other genes along paths in the network. At a stable state, the system's Jacobian matrix confers…

Biological function arises through the dynamical interactions of multiple subsystems, including those between brain areas, within gene regulatory networks, and more. A common approach to understanding these systems is to model the dynamics…

Differentiable programming techniques are widely used in the community and are responsible for the machine learning renaissance of the past several decades. While these methods are powerful, they have limits. In this short report, we…

We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an…

The edge-of-chaos dynamics of wide randomly initialized low-rank feedforward networks are analyzed. Formulae for the optimal weight and bias variances are extended from the full-rank to low-rank setting and are shown to follow from…

We introduce a novel data-driven approach aimed at designing high-quality shape deformations based on a coarse localized input signal. Unlike previous data-driven methods that require a global shape encoding, we observe that…

To make predictions or design control, information on local sensitivity of initial conditions and state-space contraction is both central, and often instrumental. However, it is not always simple to reliably determine instability fields or…

Some aspects of nonlocal dynamics on directed and undirected networks for an initial value problem whose Jacobian matrix is a variable-order fractional power of a Laplacian matrix are discussed here. This is a new extension to…

The low-dimensional manifold hypothesis posits that the data found in many applications, such as those involving natural images, lie (approximately) on low-dimensional manifolds embedded in a high-dimensional Euclidean space. In this…