Related papers: Why Deep Jacobian Spectra Separate: Depth-Induced …
We discuss several numerical methods for calculating Lyapunov exponents (a quantitative measure of chaos) in systems of ordinary differential equations. We pay particular attention to constrained systems, and we introduce a variety of…
Jacobi-type algorithms for simultaneous approximate diagonalization of real (or complex) symmetric tensors have been widely used in independent component analysis (ICA) because of their good performance. One natural way of choosing the…
When neural networks are used to model dynamics, properties such as stability of the dynamics are generally not guaranteed. In contrast, there is a recent method for learning the dynamics of autonomous systems that guarantees global…
We develop new theoretical results on matrix perturbation to shed light on the impact of architecture on the performance of a deep network. In particular, we explain analytically what deep learning practitioners have long observed…
Deep equilibrium networks (DEQs) are a new class of models that eschews traditional depth in favor of finding the fixed point of a single nonlinear layer. These models have been shown to achieve performance competitive with the…
Vulnerability of Deep Neural Networks (DNNs) to adversarial attacks has been attracting a lot of attention in recent studies. It has been shown that for many state of the art DNNs performing image classification there exist universal…
Embedding discrete solvers as differentiable layers has given modern deep learning architectures combinatorial expressivity and discrete reasoning capabilities. The derivative of these solvers is zero or undefined, therefore a meaningful…
Jacobian-vector products (JVPs) form the backbone of many recent developments in Deep Networks (DNs), with applications including faster constrained optimization, regularization with generalization guarantees, and adversarial example…
In vision science, cascades of Linear+Nonlinear transforms are very successful in modeling a number of perceptual experiences [Carandini&Heeger12]. However, the conventional literature is usually too focused on only describing the…
Deep latent variable models learn condensed representations of data that, hopefully, reflect the inner workings of the studied phenomena. Unfortunately, these latent representations are not statistically identifiable, meaning they cannot be…
This paper develops a unified analytical framework for determinant identities under finite-rank perturbations of square matrices that remains valid without invertibility assumptions. In contrast to classical inverse-based formulations, the…
The inductive biases of trained neural networks are difficult to understand and, consequently, to adapt to new settings. We study the inductive biases of linearizations of neural networks, which we show to be surprisingly good summaries of…
Works on implicit regularization have studied gradient trajectories during the optimization process to explain why deep networks favor certain kinds of solutions over others. In deep linear networks, it has been shown that gradient descent…
Large language models are remarkably capable, yet how computation propagates through their layers remains poorly understood. A growing line of work treats depth as discrete time and the residual stream as a dynamical system, where each…
Concatenating matrices is a common technique for uncovering shared structures in data through singular value decomposition (SVD) and low-rank approximations. The fundamental question arises: How does the singular value spectrum of the…
We introduce implicit Bayesian neural networks, a simple and scalable approach for uncertainty representation in deep learning. Standard Bayesian approach to deep learning requires the impractical inference of the posterior distribution…
The discrete spectrum of complex Jacobi matrices that are compact perturbations of the discrete laplacian is under consideration. The rate of stabilization for the the matrix entries which provides finiteness of the discrete spectrum and is…
We present a relative forward error analysis of a mixed-precision preconditioned one-sided Jacobi algorithm, analogous to a two-sided version introduced in [N. J. Higham, F. Tisseur, M. Webb and Z. Zhou, SIAM J. Matrix Anal. Appl. 46…
Modern neural network training relies on piece-wise (sub-)differentiable functions in order to use backpropagation to update model parameters. In this work, we introduce a novel method to allow simple non-differentiable functions at…
We study spectral properties of bounded and unbounded complex Jacobi matrices. In particular, we formulate conditions assuring that the spectrum of the studied operators is continuous on some subsets of the complex plane and we provide…