Efficient Implementation of a Higher-Order Language with Built-In AD
Abstract
We show that Automatic Differentiation (AD) operators can be provided in a dynamic language without sacrificing numeric performance. To achieve this, general forward and reverse AD functions are added to a simple high-level dynamic language, and support for them is included in an aggressive optimizing compiler. Novel technical mechanisms are discussed, which have the ability to migrate the AD transformations from run-time to compile-time. The resulting system, although only a research prototype, exhibits startlingly good performance. In fact, despite the potential inefficiencies entailed by support of a functional-programming language and a first-class AD operator, performance is competitive with the fastest available preprocessor-based Fortran AD systems. On benchmarks involving nested use of the AD operators, it can even dramatically exceed their performance.
Cite
@article{arxiv.1611.03416,
title = {Efficient Implementation of a Higher-Order Language with Built-In AD},
author = {Jeffrey Mark Siskind and Barak A. Pearlmutter},
journal= {arXiv preprint arXiv:1611.03416},
year = {2016}
}
Comments
Extended abstract presented at the AD 2016 Conference, Sep 2016, Oxford UK