Reverse-Mode AD of Reduce-by-Index and Scan in Futhark
Abstract
We present and evaluate the Futhark implementation of reverse-mode automatic differentiation (AD) for the basic blocks of parallel programming: reduce, prefix sum (scan), and reduce by index. We first present derivations of general-case algorithms and then discuss several specializations that result in efficient differentiation of most cases of practical interest. We report an experiment that evaluates the performance of the differentiated code in the context of GPU execution and highlights the impact of the proposed specializations as well as the strengths and weaknesses of differentiating at high level vs. low level (i.e., ``differentiating the memory'').
Keywords
Cite
@article{arxiv.2310.03568,
title = {Reverse-Mode AD of Reduce-by-Index and Scan in Futhark},
author = {Lotte Maria Bruun and Ulrik Stuhr Larsen and Nikolaj Hinnerskov and Cosmin Oancea},
journal= {arXiv preprint arXiv:2310.03568},
year = {2023}
}
Comments
Presented at IFL'23 (i.e., 35th Symposium on Implementation and Application of Functional Languages, Aug. 29th - 31st, 2023, Braga, Portugal)