Related papers: Matrix Calculus (for Machine Learning and Beyond)

In this paper we take a look at Automatic Differentiation through the eyes of Tensor and Operational Calculus. This work is best consumed as supplementary material for learning tensor and operational calculus by those already familiar with…

Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the implicit…

We address the task of higher-order derivative evaluation of computer programs that contain QR decompositions and real symmetric eigenvalue decompositions. The approach is a combination of univariate Taylor polynomial arithmetic and matrix…

This paper studies the expressive and computational power of discrete Ordinary Differential Equations (ODEs), a.k.a. (Ordinary) Difference Equations. It presents a new framework using these equations as a central tool for computation and…

We assume some standard choices for the branch cuts of a group of functions and consider the problem of then calculating the branch cuts of expressions involving those functions. Typical examples include the addition formulae for inverse…

Multi-task learning (MTL) optimizes several learning tasks simultaneously and leverages their shared information to improve generalization and the prediction of the model for each task. Auxiliary tasks can be added to the main task to…

In this book, there are five chapters: Systems of Linear Equations, Vector Spaces, Homogeneous Systems, Characteristic Equation of Matrix, and Matrix Dot Product. It has also exercises at the end of each chapter above to let students…

Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to…

In this work we systematically analyze general properties of differential equations used as machine learning models. We demonstrate that the gradient of the loss function with respect to to the hidden state can be considered as a…

We propose a categorical semantics of gradient-based machine learning algorithms in terms of lenses, parametrised maps, and reverse derivative categories. This foundation provides a powerful explanatory and unifying framework: it…

We derive algorithms for higher order derivative computation of the rectangular $QR$ and eigenvalue decomposition of symmetric matrices with distinct eigenvalues in the forward and reverse mode of algorithmic differentiation (AD) using…

We present in this paper some fundamental tools for developing matrix analysis over the complex quaternion algebra. As applications, we consider generalized inverses, eigenvalues and eigenvectors, similarity, determinants of complex…

Continual learning is an emerging subject in machine learning that aims to solve multiple tasks presented sequentially to the learner without forgetting previously learned tasks. Recently, many deep learning based approaches have been…

The article proposes formulating and codifying a set of applied numerical methods, coined as Deep Learning Discrete Calculus (DLDC), that uses the knowledge from discrete numerical methods to interpret the deep learning algorithms through…

Ordinary Differential Equations are generally too complex to be solved analytically. Approximations thereof can be obtained by general purpose numerical methods. However, even though accurate schemes have been developed, they remain…

Matrices are typically considered over fields or rings. Motivated by applications in parametric differential equations and data-driven modeling, we suggest to study matrices with entries from a Hilbert space and present an elementary theory…

The paper looks at a scaled variant of the stochastic gradient descent algorithm for the matrix completion problem. Specifically, we propose a novel matrix-scaling of the partial derivatives that acts as an efficient preconditioning for the…

Matrix functions are a central topic of linear algebra, and problems requiring their numerical approximation appear increasingly often in scientific computing. We review various limited-memory methods for the approximation of the action of…

Simplex gradients are an essential feature of many derivative free optimization algorithms, and can be employed, for example, as part of the process of defining a direction of search, or as part of a termination criterion. The calculation…

This is an introduction to calculus, and its applications to basic questions from physics. We first discuss the theory of functions $f:\mathbb R\to\mathbb R$, with the notion of continuity, and the construction of the derivative $f'(x)$ and…