Related papers: Forward-mode automatic differentiation for the ten…

We develop nested automatic differentiation (AD) algorithms for exact inference and learning in integer latent variable models. Recently, Winner, Sujono, and Sheldon showed how to reduce marginalization in a class of integer latent variable…

Adaptive optimizers, such as Adam, have achieved remarkable success in deep learning. A key component of these optimizers is the so-called preconditioning matrix, providing enhanced gradient information and regulating the step size of each…

We address the task of higher-order derivative evaluation of computer programs that contain QR decompositions and real symmetric eigenvalue decompositions. The approach is a combination of univariate Taylor polynomial arithmetic and matrix…

Computing free energy is a fundamental problem in statistical physics. Recently, two distinct methods have been developed and have demonstrated remarkable success: the tensor-network-based contraction method and the neural-network-based…

This work proposes a time-efficient Natural Gradient Descent method, called TENGraD, with linear convergence guarantees. Computing the inverse of the neural network's Fisher information matrix is expensive in NGD because the Fisher matrix…

The higher-order tensor renormalization group is a tensor-network method providing estimates for the partition function and thermodynamical observables of classical and quantum systems in thermal equilibrium. At every step of the iterative…

We propose a method to compute the entanglement entropy (EE) using the tensor renormalization group (TRG) method. The reduced density matrix of a $d$-dimensional quantum system is represented as a $(d+1)$-dimensional tensor network. We…

Automatic generation of convex relaxations and subgradients is critical in global optimization, and is typically carried out using variants of automatic/algorithmic differentiation (AD). At previous AD conferences, variants of the forward…

Automatic differentiation (AD) has driven recent advances in machine learning, including deep neural networks and Hamiltonian Markov Chain Monte Carlo methods. Partially observed nonlinear stochastic dynamical systems have proved resistant…

How does one compile derivatives of tensor programs, such that the resulting code is purely functional (hence easier to optimize and parallelize) and provably efficient relative to the original program? We show that naively differentiating…

We present a novel acceleration technique for improving the convergence of source iteration for discrete ordinates transport calculations. Our approach uses the idea of the dynamic mode decomposition (DMD) to estimate the slowly decaying…

Algorithmic differentiation (AD) has become increasingly capable and straightforward to use. However, AD is inefficient when applied directly to solvers, a feature of most engineering analyses. We can leverage implicit differentiation to…

Second-order optimization techniques have the potential to achieve faster convergence rates compared to first-order methods through the incorporation of second-order derivatives or statistics. However, their utilization in deep learning is…

We introduce the order-separated Grassmann higher-order tensor renormalization group (OS-GHOTRG) method for QCD with staggered quarks in the strong-coupling expansion. The method allows us to determine the expansion coefficients of the…

Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives…

Recent advances in neural-network architecture allow for seamless integration of convex optimization problems as differentiable layers in an end-to-end trainable neural network. Integrating medium and large scale quadratic programs into a…

We combine the multigrid (MG) method with state-of-the-art concepts from the variational formulation of the numerical renormalization group. The resulting MG renormalization (MGR) method is a natural generalization of the MG method for…

In usual (non-stochastic) tensor network calculations, the truncated singular value decomposition (SVD) is often used for approximating a tensor, and it causes systematic errors. By introducing stochastic noise in the approximation,…

Parametric manifold optimization problems frequently arise in various machine learning tasks, where state functions are defined on infinite-dimensional manifolds. We propose a unified accelerated natural gradient descent (ANGD) framework to…

We present a GPU-based system for automatic differentiation (AD) of functions defined on triangle meshes, designed to exploit the locality and sparsity in mesh-based computation. Our system evaluates derivatives using per-element…