Related papers: Matrix Calculus (for Machine Learning and Beyond)
This paper is a broad and accessible survey of the methods we have at our disposal for Monte Carlo gradient estimation in machine learning and across the statistical sciences: the problem of computing the gradient of an expectation of a…
The standard text book theory of ODEs lacks a general method to solve linear equations having variable coefficients, providing instead a collection of special techniques for particular classes of equations. The present article addresses…
This paper presents a comprehensive review of loss functions and performance metrics in deep learning, highlighting key developments and practical insights across diverse application areas. We begin by outlining fundamental considerations…
Electrical circuits are present in a variety of technologies, making their design an important part of computer aided engineering. The growing number of parameters that affect the final design leads to a need for new approaches to quantify…
A well-known problem in computing some matrix functions iteratively is the lack of a clear, commonly accepted residual notion. An important matrix function for which this is the case is the matrix exponential. Suppose the matrix exponential…
Embedding Calculus, as described by Weiss, is a calculus of functors, suitable for studying contravariant functors from the poset of open subsets of a smooth manifold M, denoted O(M), to a category of topological spaces (of which the…
Automatic differentiation is involved for long in applied mathematics as an alternative to finite difference to improve the accuracy of numerical computation of derivatives. Each time a numerical minimization is involved, automatic…
The purpose of this text is to provide an accessible introduction to a set of recently developed algorithms for factorizing matrices. These new algorithms attain high practical speed by reducing the dimensionality of intermediate…
The reverse derivative is a fundamental operation in machine learning and automatic differentiation. This paper gives a direct axiomatization of a category with a reverse derivative operation, in a similar style to that given by Cartesian…
The derivation of a function is a fundamental tool for solving problems in calculus. Consequently, the motivations for investigating physical systems capable of performing this task are numerous. Furthermore, the potential to develop an…
The notion of a Moreau envelope is central to the analysis of first-order optimization algorithms for machine learning. Yet, it has not been developed and extended to be applied to a deep network and, more broadly, to a machine learning…
This book intends to give the main definitions and theorems in mathematics which could be useful for workers in theoretical physics. It gives an extensive and precise coverage of the subjects which are addressed, in a consistent and…
Objective functions that optimize deep neural networks play a vital role in creating an enhanced feature representation of the input data. Although cross-entropy-based loss formulations have been extensively used in a variety of supervised…
We study a general class of bilevel problems, consisting in the minimization of an upper-level objective which depends on the solution to a parametric fixed-point equation. Important instances arising in machine learning include…
The primary goal of this paper is to introduce and investigate generalized incomplete exponential functions with matrix parameters. Integral representation, differential formula, addition formula, multiplication formula, and recurrence…
This paper provides a new approach to derive various arbitrary high order finite difference formulae for the numerical differentiation of analytic functions. In this approach, various first and second order formulae for the numerical…
We develop a compositional approach for automatic and symbolic differentiation based on categorical constructions in functional analysis where derivatives are linear functions on abstract vectors rather than being limited to scalars,…
Supervised classification and representation learning are two widely used classes of methods to analyze multivariate images. Although complementary, these methods have been scarcely considered jointly in a hierarchical modeling. In this…
In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to…
This paper surveys the machine learning literature and presents in an optimization framework several commonly used machine learning approaches. Particularly, mathematical optimization models are presented for regression, classification,…