CHAD: Combinatory Homomorphic Automatic Differentiation
share
We introduce Combinatory Homomorphic Automatic Differentiation (CHAD), a principled, pure, provably correct method for performing forward- and reverse-mode automatic differentiation (AD) on programming languages with expressive features. It implements AD as a compositional, type-respecting source-code transformation that generates purely functional code. This code transformation is principled in the sense that it is the unique homomorphic (structure preserving) extension to expressive languages of the well-known and unambiguous definitions of automatic differentiation for a first-order functional language. Correctness of the method follows by a (compositional) logical relations argument that shows that the semantics of the syntactic derivative is the usual calculus derivative of the semantics of the original program. In their most elegant formulation, the transformations generate code with linear types. However, the transformations can be implemented in a standard functional language without sacrificing correctness. This implementation can be achieved by making use of abstract data types to represent the required linear types, e.g. through the use of a basic module system. In this paper, we detail the method when applied to a simple higher-order language for manipulating statically sized arrays. However, we explain how the methodology applies, more generally, to functional languages with other expressive features. Finally, we discuss how the scope of CHAD extends beyond applications in automatic differentiation to other dynamic program analyses that accumulate data in a commutative monoid.
READ FULL TEXT