Related papers: Backpropagation through Back Substitution with a B…

In problems of mathematical physics, to study the structures of spaces using the Cayley-Klein models in theoretical calculations, the use of generalized complex numbers is required. In the case of computational experiments, such tasks…

Gaussian filters have applications in a variety of areas in computer science, from computer vision to speech recognition. The collapsing sum is a matrix operator that was recently introduced to study Gaussian filters combinatorially. In…

Second-order optimization uses curvature information about the objective function, which can help in faster convergence. However, such methods typically require expensive computation of the Hessian matrix, preventing their usage in a…

We introduce backdrop, a flexible and simple-to-implement method, intuitively described as dropout acting only along the backpropagation pipeline. Backdrop is implemented via one or more masking layers which are inserted at specific points…

This paper is addressed to the well-posedness of some linear and semilinear backward stochastic differential equations with general filtration, without using the Martingale Representation Theorem. The point of our approach is to introduce a…

In this paper we describe a parallel Gaussian elimination algorithm for matrices with entries in a finite field. Unlike previous approaches, our algorithm subdivides a very large input matrix into smaller submatrices by subdividing both…

With a view towards models of quantum computation and/or the interpretation of linear logic, we define a functional language where all functions are linear operators by construction. A small step operational semantic (and hence an…

The backpropagation algorithm for neural networks is widely felt hard to understand, despite the existence of some well-written explanations and/or derivations. This paper provides a new derivation of this algorithm based on the concept of…

Composite function minimization captures a wide spectrum of applications in both computer vision and machine learning. It includes bound constrained optimization, $\ell_1$ norm regularized optimization, and $\ell_0$ norm regularized…

We show that the forward and backward propagation can be formulated as a solution of lower and upper triangular systems of equations. For standard feedforward (FNNs) and recurrent neural networks (RNNs) the triangular systems are always…

We introduce a new technique for gradient normalization during neural network training. The gradients are rescaled during the backward pass using normalization layers introduced at certain points within the network architecture. These…

Gaussian processes are a class of flexible nonparametric Bayesian tools that are widely used across the sciences, and in industry, to model complex data sources. Key to applying Gaussian process models is the availability of well-developed…

In a variety of problems originating in supervised, unsupervised, and reinforcement learning, the loss function is defined by an expectation over a collection of random variables, which might be part of a probabilistic model or the external…

In this note, we propose a simple-looking but broad conjecture about star-algebras over the field of real numbers. The conjecture enables many matrix decompositions to be represented by star-algebras and star-ideals. This paper is written…

In distributed optimization and distributed numerical linear algebra, we often encounter an inversion bias: if we want to compute a quantity that depends on the inverse of a sum of distributed matrices, then the sum of the inverses does not…

Unitary neural networks are promising alternatives for solving the exploding and vanishing activation/gradient problem without the need for explicit normalization that reduces the inference speed. However, they often require longer training…

We develop theory and software for rotation equivariant operators on scalar and vector fields, with diverse applications in simulation, optimization and machine learning. Rotation equivariance (covariance) means all fields in the system…

The Cartesian reverse derivative is a categorical generalization of reverse-mode automatic differentiation. We use this operator to generalize several optimization algorithms, including a straightforward generalization of gradient descent…

A new totally algebraic formalism based on general, abstract ladder operators has been proposed. This approach heavily grounds in the superoperator formalism of Primas. However it is necessary to introduce many improvements in his…

Inspired by Gauss-Newton-like methods, we study the benefit of leveraging the structure of deep learning objectives, namely, the composition of a convex loss function and of a nonlinear network, in order to derive better direction oracles…