Related papers: Matrix Calculus (for Machine Learning and Beyond)
The dynamic matrix inverse problem is to maintain the inverse of a matrix undergoing element and column updates. It is the main subroutine behind the best algorithms for many dynamic problems whose complexity is not yet well-understood,…
We give an algorithm to compute the series expansion for the inverse of a given function. The algorithm is extremely easy to implement and gives the first $N$ terms of the series. We show several examples of its application in calculating…
Results of research of possibility of transformation of a difference equation into a system of the first-order difference equation are presented. In contrast to the method used previously, an unknown grid function is split into two new…
A method to increase the precision of feedforward networks is proposed. It requires a prior knowledge of a target function derivatives of several orders and uses this information in gradient based training. Forward pass calculates not only…
Derivatives of computer graphics, image processing, and deep learning algorithms have tremendous use in guiding parameter space searches, or solving inverse problems. As the algorithms become more sophisticated, we no longer only need to…
Random matrices now play a role in many parts of computational mathematics. To advance these applications, it is desirable to have tools that are flexible, easy to use, and powerful. Over the last 25 years, researchers have developed a…
A classification of commutative integral domains consisting of ordinary differential operators with matrix coefficients is established in terms of morphisms between algebraic curves.
Incremental computation aims to compute more efficiently on changed input by reusing previously computed results. We give a high-level overview of works on incremental computation, and highlight the essence underlying all of them, which we…
Backpropagation is a classic automatic differentiation algorithm computing the gradient of functions specified by a certain class of simple, first-order programs, called computational graphs. It is a fundamental tool in several fields, most…
Recent theoretical work on automatic differentiation (autodiff) has focused on characteristics such as correctness and efficiency while assuming that all derivatives are automatically generated by autodiff using program transformation, with…
A new directional derivative and a new subdifferential for set-valued convex functions are constructed, and a set-valued version of the so-called 'max-formula' is proven. The new concepts are used to characterize solutions of convex…
We study a class of matrix function algebras, here denoted $\mathcal{T}^{+}(\mathcal{C}_n)$. We introduce a notion of point derivations, and classify the point derivations for certain finite dimensional representations of…
Partition functions of eigenvalue matrix models possess a number of very different descriptions: as matrix integrals, as solutions to linear and non-linear equations, as tau-functions of integrable hierarchies and as special-geometry…
These lecture notes for a graduate course cover generalized derivative concepts useful in deriving necessary optimality conditions and numerical algorithms for nondifferentiable optimization problems in inverse problems, imaging, and…
In this article we describe special type of mathematical problems that may help develop teaching methods that motivate students to explore patterns, formulate conjectures and find solutions without only memorizing formulas and procedures.…
Neural Ordinary Differential Equations (Neural ODEs) represent a significant breakthrough in deep learning, promising to bridge the gap between machine learning and the rich theoretical frameworks developed in various mathematical fields…
The proper handling of 3D orientations is a central element in many optimization problems in engineering. Unfortunately many researchers and engineers struggle with the formulation of such problems and often fall back to suboptimal…
Recent years have witnessed a growth in mathematics for deep learning--which seeks a deeper understanding of the concepts of deep learning with mathematics and explores how to make it more robust--and deep learning for mathematics, where…
This book aims to provide an introduction to the topic of deep learning algorithms. We review essential components of deep learning algorithms in full mathematical detail including different artificial neural network (ANN) architectures…
We propose a general technique for improving alternating optimization (AO) of nonconvex functions. Starting from the solution given by AO, we conduct another sequence of searches over subspaces that are both meaningful to the optimization…