Automatic Differentiation - A Revisionist History and the State of the Art - AD meets SDG and PLT

Опубликовано 8 июля 2016, 0:23

Automatic Differentiation - A Revisionist History and the State of the Art (hour 1) AD meets SDG and PLT (hour 2)
Automatic Differentiation (aka Algorithmic Differentiation, aka Computational Differentiation, aka AD) is an established discipline concerning methods of transforming algorithmic processes (ie, computer programs) which calculate numeric functions to also calculate various derivatives of interest, and ways of using such methods. We begin with a discussion of the venerable history of the field, whose roots go back to the dawn of the computer age. There are various "modes" of automatic differentiation, and we will describe forward, reverse, and checkpoint-reverse modes in detail. We then turn our attention to existing systems, with a focus on those able to attain high performance, and the implementation techniques which allow this. These implementations have a variety of weaknesses and restrictions which, we will argue, have impeded the uptake of AD. We close with a benchmark comparing the fastest current systems with our own research prototype compiler based on principles discussed in Part Two. AD is generally cast as transforming first-order code which consumes and produces numeric arrays. We open with a laundry list of practical weaknesses of this standard formulation and standard implementation technologies. We proceed to a discussion of attempts to generalize AD: to make it apply to a wider class of procedures, to raise its semantic level, to increase its robustness, and to make it first-class. We sketch out links to Synthetic Differential Geometry and to Programming Language Theory, and summarize a few ways in which AD can be formalized. Refactoring of AD opens the possibility of using techniques being developed for efficient AD to serve other ends, such as security monitoring or probabilistic computation. We close with some case studies of ways in which general AD, combined with extremely aggressive compilation, can allow highly succinct and modular construction of complex adaptive systems. (Joint work with Jeffrey Mark Siskind.)