Language

We consider an idealized functional language with product types $\tau \,{\mathop{\times }}\,\sigma$ , sum types $\tau \,{\mathop{\sqcup }}\,+\sigma$ , and function types $\tau \to \sigma$ generated by

• • a primitive type $\mathbf{real}$ of real numbers (in practice, implemented as floating-point numbers);

• • constants $\vdash \underline{c}:\mathbf{real}$ for $c\in \mathbb{R}$ ;

• • sets $(\mathrm{Op}_n)_{n\in \mathbb{N}}$ of $n$ -ary primitive operations $\mathrm{op}$ , for which we include computations $x_1:\mathbf{real},\ldots, x_n:\mathbf{real}\vdash \mathrm{op} (x_1,\ldots, x_n):\mathbf{real}$ ; we think of these as implementing partial functions $\mathbb{R}^n\rightharpoonup \mathbb{R}$ with open domain of definition, on which they are differentiable; for example, we can include mathematical operations $\log, \exp \in \mathrm{Op}_1$ and $(+),(\!*\!),(/)\in \mathrm{Op}_2$ ;

• • a construct $x:\mathbf{real}\vdash \mathbf{sign}\,(x):\mathbf{1}\,{\mathop{\sqcup }}\,+\mathbf{1}$ , where we write $\mathbf{1}$ for the empty product; $\mathbf{sign}\,{t}$ computes the sign of a real number $t$ and is undefined at ${t}=\underline{0}$ ; we can use it to define a lazy conditional on real numbers $\mathbf{if}\,r\,\mathbf{then}\,t\,\mathbf{else}\,s\, \stackrel{\mathrm{def}}{=} \mathbf{case}\,{\mathbf{sign}\,{r}}\;\mathbf{of}{\{\_\to{t}\mathrel{\big \lvert } \_\to{r}\}}$ of the kind that is often used in AD libraries like Stan (Carpenter et al. Reference Carpenter, Hoffman, Brubaker, Lee, Li and Betancourt2015).

Next, we include two more standard mechanisms for defining partial functions:

• • (purely functional) iteration: given a computation $\Gamma, x \;:\; \tau \vdash{t} \;:\; \tau \,{\mathop{\sqcup }}\,+\sigma$ to iterate and a starting value $\Gamma \vdash{s} \;:\; \tau$ , we have a computation $\Gamma \vdash \mathbf{iterate}\,{t}\,\mathbf{from}\,{x}={s} \;:\; \sigma$ , which repeatedly calls $t$ , starting from the value of $s$ until the result lies in $\sigma$ ;

• • recursion: given a computation $\Gamma, x:\tau \to \sigma \vdash{t}:\tau \to \sigma$ , we have a program $\Gamma \vdash \mu{x}.{t}:\tau\,{\rightarrow }\,\sigma$ that recursively computes to $\mathbf{let}\,x=\,\mu x.t\,\mathbf{in}\,t$ ; note that we can define iteration with recursion.

Dual numbers forward AD code transform

Let us assume that we have programs $\partial _i\mathrm{op} (x_1,\ldots, x_n)$ that compute the $i$ -th partial derivative of each $n$ -ary primitive operation $\mathrm{op}$ . For example, we can define $\partial _1(\!*\!)(x_1,x_2)=x_2$ and $\partial _2(\!*\!)(x_1,x_2)=x_1$ . Then, we can define a very straightforward forward mode AD code transformation $\mathcal{D}$ by replacing all primitive types $\mathbf{real}$ by a pair $\mathcal{D}\;({\mathbf{real}})\stackrel{\mathrm{def}}{=} \mathbf{real}\,{\mathop{\times }}\,\mathbf{real}$ of reals and by replacing all constants $\underline{c}$ , $n$ -ary primitive operations $\mathrm{op}$ and sign function $\mathbf{sign}\,$ in the program asFootnote ²

\begin{equation*} \begin {array}{ll} \mathcal {D}\;(\underline {c}) \stackrel {\mathrm {def}}{=} & \langle {\underline {c}},{\underline {0}}\rangle\\ \mathcal {D}\;(\mathrm {op} ({r}_1,\ldots, {r}_n))\stackrel {\mathrm {def}}{=}\, &\mathbf {case}\,\mathcal {D}\;{({r}_1)}\,\mathbf {of}\,\langle {x_1},{x^{\prime}_1}\rangle \rightarrow \ldots \,{\rightarrow }\,\mathbf {case}\,{\mathcal {D}\;{({r}_n)}}\,\mathbf {of}\,\langle {x_n},{x^{\prime}_n}\,\rangle \rightarrow \\ &{\langle { \mathrm {op} (x_1,\ldots, x_n)},{x^{\prime}_1 *\partial _1\mathrm {op} (x_1,\ldots, x_n)+\ldots +x^{\prime}_n *\partial _n\mathrm {op} (x_1,\ldots, x_n)}\rangle }\\ \mathcal {D}\;{(\mathbf {sign}\,{r})}\stackrel {\mathrm {def}}{=} & \mathbf {sign}\,{(\mathbf {fst}\,\mathcal {D}\;{(r)})}. \end {array} \end{equation*}

We extend $\mathcal{D}$ to all other types and programs in the unique homomorphic (structure preserving way), by using structural recursion. So, for example, $\mathcal{D}\;{(\tau \to \sigma )}\stackrel{\mathrm{def}}{=} \mathcal{D}\;{(\tau )}\to \mathcal{D}\;{(\sigma )}$ , $\mathcal{D}\;{(x)}\stackrel{\mathrm{def}}{=} x$ , $\mathcal{D}\;{(\mathbf{let}\,x=\,t\,\mathbf{in}\,s)}={\mathbf{let}\,x} ={\mathcal{D}\;{(t)}} \,\mathbf{in}\,{\mathcal{D}\;{(s)}}$ , and $\mathcal{D}\;{({t}\,{s})}=\mathcal{D}\;{(t)}\,\mathcal{D}\;{(s)}$ . We like to think of $\mathcal{D}$ as a structure preserving functor $\mathcal{D}:\mathbf{Syn}\,{\rightarrow }\,\mathbf{Syn}$ on the syntax.

Semantics

To formulate correctness of the AD transformation $\mathcal{D}$ , we need to assign a formal denotational semantics $[\![-]\!]$ to our language. We use the standard interpretation of types $\tau$ as $\omega$ -cpos $[\![\tau ]\!]$ (partially ordered sets with suprema of countable chains) and programs $x_1:\tau _1,\ldots, x_n:\tau _n\vdash{t}:\sigma$ as monotone $\omega$ -continuous partial functions $[\![t]\!] :[\![\tau _1]\!] \times \cdots \times [\![\tau _n]\!] \rightharpoonup [\![\sigma ]\!]$ . We interpret $\mathbf{real}$ as the discrete $\omega$ -cpo $[\![\mathbf{real}]\!] \stackrel{\mathrm{def}}{=} \mathbb{R}$ of real numbers, in which $r\leq r'$ if and only if $r=r'$ . We interpret $\underline{c}$ as the constant $[\![\underline{c}]\!] \stackrel{\mathrm{def}}{=} c\in \mathbb{R}$ . We interpret $\mathrm{op}$ as the partial differentiable function $[\![\mathrm{op} (x_1,\ldots, x_n)]\!] :\mathbb{R}^n\rightharpoonup \mathbb{R}$ that it is intended to implement. And, finally, we interpret $\mathbf{sign}\,$ as the partial function $[\![\mathbf{sign}\,(x)]\!] :\mathbb{R}\rightharpoonup \mathbf{1} \sqcup \mathbf{1}$ that sends $r\lt 0$ to the left copy of $\mathbf{1}$ and $r\gt 0$ to the right copy and is undefined for $r=0$ . Having fixed these definitions, the rest of the semantics is entirely compositional and standard. In particular, we interpret iteration and recursion using Kleene’s fixpoint theorem. We think of this semantics as a structure preserving functor $[\![-]\!] :\mathbf{Syn}\,{\rightarrow }\,\boldsymbol{\omega } \mathbf{Cpo}$ from the syntax to the category of $\omega$ -cpos and monotone $\omega$ -continuous functions.

Correctness statement

Having defined a semantics, we can phrase what it means for $\mathcal{D}$ to be correct. We prove the following, showing that $\mathcal{D}\;{(t)}$ implements the usual calculus derivative $D[\![t]\!]$ of $[\![t]\!]$ .

Theorem 2.1 (Forward AD correctness, Theorem 9.1 with $k=1$ in main text). For any program $x:\tau \vdash{t}:\sigma$ for $\tau =\mathbf{real}^{m},\sigma =\mathbf{real}^l$ (where we write $\mathbf{real}^n$ for the type $\mathbf{real}\,{\mathop{\times }}\,\cdots \,{\mathop{\times }}\,\mathbf{real}$ of length $n$ tuples of reals), we have that $[\![t]\!]$ is differentiable on its domain and

\begin{align*} &[\![\mathcal{D}\;{(t)}]\!] ((x_{1},v_{1}),\ldots, (x_{m},v_{m}))=\\ &\Big (\pi _1([\![t]\!] (x_1,\ldots, x_m)), \pi _1(D[\![t]\!] ((x_{1},\ldots, x_m),(v_{1},\ldots, v_m))),\ldots, \\ &\;\;\pi _l([\![t]\!] (x_1,\ldots, x_m)), \pi _l(D[\![t]\!] ((x_{1},\ldots, x_m),(v_{1},\ldots, v_m))) \Big ) \end{align*}

for any $(x_1,\ldots, x_m)$ in the domain of definition of $[\![t]\!]$ and any tangent vector $(v_1,\ldots, v_m)$ to $[\![\tau ]\!]$ at $x$ .

Importantly, the program $t$ might use higher-order functions, iteration, recursion, etc. In fact, we also establish the theorem above for general types $\tau$ and $\sigma$ not containing function types, but its phrasing requires slight bookkeeping that might distract from the simplicity of the theorem.

A proof via logical relations

The proof of the correctness theorem follows a logical relations argument that we found using categorical methods, but which can be phrased entirely in elementary terms. Let us fix some $n\in \mathbb{N}$ . We define for all types $\tau$ of our language, by induction, relations $T^{n}_{\tau }\subseteq (\mathbb{R}^n\to [\![\tau ]\!] )\times ((\mathbb{R}^n\times \mathbb{R}^n)\to [\![\mathcal{D}\;{(\tau )}]\!] )$ and $P^{n}_{\tau }\subseteq (\mathbb{R}^n\rightharpoonup{[\![\tau ]\!] })\times ((\mathbb{R}^n\times \mathbb{R}^n)\rightharpoonup{[\![\mathcal{D}\;{(\tau )}]\!] })$ that relate a (partial) $n$ -curve to its derivative $n$ -curve:

\begin{align*} T^{n}_{\mathbf{real}}&\stackrel{\mathrm{def}}{=} \{(\gamma, \gamma ')\mid \gamma \text{ is differentiable and } \gamma '=(x,v)\mapsto (\gamma (x),D\gamma (x,v))\}\\ T^{n}_{\tau \,{\mathop{\times }}\,\sigma } & \stackrel{\mathrm{def}}{=} \{(x\mapsto (\gamma _1(x),\gamma _2(x)), (x,v)\mapsto (\gamma^{\prime}_1(x,v),\gamma^{\prime}_2(x,v)))\mid (\gamma _1,\gamma^{\prime}_1)\in T^{n}_{\tau }\text{ and } (\gamma _2,\gamma^{\prime}_2)\in T^{n}_{\sigma }\}\\ T^{n}_{\tau \,{\mathop{\sqcup }}\,+\sigma } & \stackrel{\mathrm{def}}{=} \{(\iota _1\circ \gamma _1,\iota _1\circ \gamma^{\prime}_1)\mid (\gamma _1,\gamma^{\prime}_1)\in T^{n}_{\tau }\}\cup \{(\iota _2\circ \gamma _2,\iota _2\circ \gamma^{\prime}_2)\mid (\gamma _2,\gamma^{\prime}_2)\in T^{n}_{\sigma }\}\\ T^{n}_{\tau \to \sigma }& \stackrel{\mathrm{def}}{=} \{(\gamma, \gamma ')\mid \forall (\delta, \delta ')\in T^{n}_{\tau }. (x\mapsto \gamma (x)(\delta (x)), (x,v)\mapsto \gamma '(x,v)(\delta '(x,v)))\in P^{n}_{\sigma }\}\\ P^{n}_{\tau } & \stackrel{\mathrm{def}}{=} \Big \{(\gamma, \gamma ')\mid \gamma ^{-1}([\![\tau ]\!] )\times \mathbb{R}^n=\gamma ^{'-1}([\![\mathcal{D}\;{(\tau )}]\!] )\text{ is open and for all differentiable}\\&\qquad \qquad \quad \delta :\mathbb{R}^n\,{\rightarrow }\, \gamma ^{-1}([\![\tau ]\!] ) \text{ we have } (\gamma \circ \delta, (x,v)\mapsto (\gamma (\delta (x)),\gamma '(D\delta (x,v)))) \in T^{n}_{\tau }\Big \}. \end{align*}

We then prove the following “fundamental lemma,” using induction on the typing derivation of $t$ :

For example, we use that, by assumption, $[\![\partial _i\mathrm{op} (x_1,\ldots, x_n)]\!]$ equals the $i$ -th partial derivative of $[\![\mathrm{op} (x_1,\ldots, x_n)]\!]$ combined with the chain rule, to show that primitive operations $\mathrm{op}$ respect the logical relations. Crucial features to enable the inductive steps for iteration and recursion in the proof of the fundamental lemma are that $T^n_{\mathbf{real}}$ and $P^n_{\tau }$ are closed under suprema of countable chains and that $P^n_{\tau }$ contains the least element.

As $T^{k}_{\mathbf{real}^k}$ contains, in particular, $(\textrm{id}, ((x_1,\ldots, x_k), (v_1,\ldots, v_k))\mapsto ((x_1,v_1),\ldots, (x_n,v_k)))$ , our correctness theorem follows.

Extending to recursive types via a novel categorical logical relations technique

Next, we extend our language with ML-style polymorphism and recursive types. That is, we allow the formation of types $\tau$ with free type variables $\alpha$ , and we include a type variable binder $\boldsymbol{\mu }\alpha .\tau$ , which binds $\alpha$ in $\tau$ and computes a canonical fixpoint of $\alpha \mapsto \tau$ . We extend our AD transformation homomorphically on terms and types. For example, on types, we define

\begin{align*} \mathcal{D}\;{(\alpha )}\stackrel{\mathrm{def}}{=}\alpha && \mathcal{D}\;{(\boldsymbol{\mu }\alpha .\tau )}\stackrel{\mathrm{def}}{=}\boldsymbol{\mu }\alpha .\mathcal{D}\;{(\tau )}. \end{align*}

A type $\tau$ with $n$ free type variables gets interpreted in our $\omega$ -cpo-semantics as an $n$ -ary mixed-variance endofunctor $[\![\tau ]\!]$ on the category of $\omega$ -cpos and partial morphisms that restricts to that of $\omega$ -cpos and total morphisms. Programs with types that have free variables get interpreted as (di)natural transformations. As the category of $\omega$ -cpos and partial morphisms has the structure to interpret recursive types, we have a canonical minimal invariant

\begin{equation*} \mathrm {roll}:[\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}] (\mu [\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}], \mu [\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}] )\xrightarrow {\,{\cong}\, }\mu [\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}] \end{equation*}

for the mixed-variance endofunctors $[\![\tau ]\!]$ on $\boldsymbol{\omega } \mathbf{Cpo}$ that types $\tau$ denote (Levy Reference Levy2012). We interpret $[\![\boldsymbol{\mu }\alpha .\tau ]\!] \stackrel{\mathrm{def}}{=} \mu [\![\tau ]\!]$ .

To extend the correctness proof to this larger language, we would like to define the logical relation

\begin{equation*} T^n_{\boldsymbol{\mu }\alpha .\tau }\stackrel {\mathrm {def}}{=} \{(\mathrm {roll}\circ \gamma, \mathrm {roll}\circ \gamma ')\mid (\gamma, \gamma ')\in T^n_{\tau {}[^{\boldsymbol{\mu }\alpha .\tau }\!/\!_{\alpha }]}\}. \end{equation*}

That is, we would like to be able to define relations using type recursion. If we can do so, then extending the proof of the fundamental lemma is straightforward. We can then establish the correctness theorem also for $\tau$ and $\sigma$ that involve recursive types.

The traditional method is to follow the technical recipes of Pitts (Reference Pitts1996). Instead, we develop a powerful new logical relations technique for recursive types, which we believe to be more conceptually clear and easier to use in situations like ours. To be precise, we prove a general result saying that under mild conditions, we can interpret recursive types in the category of logical relations over a category that models recursive types itself. For simplicity, we state an important special case that we need for our application here.

Given a right adjoint $\boldsymbol{\omega } \mathbf{Cpo}$ -enriched functor $G:\boldsymbol{\omega } \mathbf{Cpo}^n\,{\rightarrow }\, \boldsymbol{\omega } \mathbf{Cpo}$ (such as $G(X)=\boldsymbol{\omega } \mathbf{Cpo}^n(Y, X)$ for some $Y\in \boldsymbol{\omega } \mathbf{Cpo}^n$ ), consider the category $\mathbf{SScone}$ of $n$ -ary logical relations, which has objects $(X, T)$ , where $X\in \boldsymbol{\omega } \mathbf{Cpo}^n$ and $T$ is a (full) sub- $\omega$ -cpo of $GX$ , and morphisms $(T,X)\,{\rightarrow }\, (T',X')$ are $\boldsymbol{\omega } \mathbf{Cpo}^n$ -morphisms $f:X\,{\rightarrow }\, X'$ such that $y\in T$ implies $Gf(y)\in T'$ .

Spelled out in non-categorical terms, we are considering logical relations

\begin{equation*} T_{\tau }\subseteq G([\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}] )\qquad \qquad P_{\tau }=L(T_{\tau },[\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}] )\subseteq G({[\hspace {-2.5pt}[\tau ]\hspace {-2.5pt}] }_{\bot }) \end{equation*}

and we require that the relation $P_{0}$ at the initial object (empty type) is precisely the singleton set $\{\bot \}$ (containing the least element) and $G({[\![\tau ]\!] }\subseteq{{[\![\tau ]\!] }}_{\bot })^{-1}(P_{\tau })$ (which we think of as the total elements in $P_{\tau }$ ) coincide with $T_{\tau }$ .

In particular, in our case, we work binary logical relations ( $n=2$ ) with

\begin{equation*}G(X,X')=\boldsymbol {\omega } \mathbf {Cpo}^2((\mathbb {R}^n,\mathbb {R}^n\times \mathbb {R}^n), (X,X'))\end{equation*}

and the monad lifting

\begin{align*} L(T,(X,X')) &= \Big \{(\gamma, \gamma ')\mid \gamma ^{-1}(X)\times \mathbb{R}^n=\gamma ^{'-1}(X')\text{ is a proper open subset and for all differentiable}\\&\qquad \qquad \quad \delta :\mathbb{R}^n\,{\rightarrow }\, \gamma ^{-1}(X) \text{ we have } (\gamma \circ \delta, (x,v)\mapsto (\gamma (\delta (x)),\gamma '(D\delta (x,v)))) \in T\Big \}\\ &\phantom{==}\cup T. \end{align*}

Consequently, we can define the logical relations $T_{\boldsymbol{\mu }\alpha .\tau }$ using type recursion, as desired.

Dual numbers reverse AD

Similarly to dual numbers forward AD $\mathcal{D}$ , we can define a reverse AD code transformation $\overleftarrow{\mathcal{D}}$ : we define $ \overleftarrow{\mathcal{D}}(\mathbf{real})\stackrel{\mathrm{def}}{=} \mathbf{real}\,{\mathop{\times }}\,\mathbf{vect}$ and

\begin{align*} \overleftarrow{\mathcal{D}}(\underline{c}) &\stackrel{\mathrm{def}}{=} \langle{\underline{c}},{{ 0}^{{\mathbf{v}}}}\rangle \\ \overleftarrow{\mathcal{D}}({\mathrm{op} ({t}_1,\ldots, {t}_n)}) &\stackrel{\mathrm{def}}{=} \mathbf{case}\,{\overleftarrow{\mathcal{D}}({{t}_1})}\,\mathbf{of}\,\langle{x_1},{x^{\prime}_1}\rangle \to{\ldots }\; \mathbf{case}\,{\overleftarrow{\mathcal{D}}({{t}_n})}\,\mathbf{of}\,\langle{x_n},{x^{\prime}_n}\rangle \to \\ &\quad \quad \langle \mathrm{op} (x_1,\ldots, x_n),{x^{\prime}_1}*^{{\mathbf{v}}}{\partial _1\mathrm{op} (x_1,\ldots, x_n)}{+}^{\mathbf{v}}\ldots{+}^{\mathbf{v}}{x^{\prime}_n}*^{{\mathbf{v}}}{\partial _n\mathrm{op} (x_1,\ldots, x_n)} \rangle \\ \overleftarrow{\mathcal{D}}{(\mathbf{sign}\,{t})} &\stackrel{\mathrm{def}}{= } \mathbf{sign}\,(\mathbf{fst}\,\overleftarrow{\mathcal{D}}({t})). \end{align*}

and extend homomorphically to all other type and term formers, as we did before. In fact, this algorithm is exactly the same as dual numbers forward AD in code with the only differences being that

1. (1) the type $\mathbf{real}$ of real numbers for tangents has been replaced with a new type $\mathbf{vect}$ , which we think of as representing (dynamically sized) cotangent vectors to the global input of the program;

2. (2) the zero $\underline{0}$ and addition $(+)$ of type $\mathbf{real}$ have been replaced by the zero ${ 0}^{{\mathbf{v}}}$ and addition $({+}^{\mathbf{v}})$ of cotangents of type $\mathbf{vect}$ ;

3. (3) the multiplication $(\!*\!):\mathbf{real}\,{\mathop{\times }}\,\mathbf{real}\,{\rightarrow }\, \mathbf{real}$ has been replaced by the operation $(*^{{\mathbf{v}}}):\mathbf{vect} \,{\mathop{\times }}\, \mathbf{real}\,{\rightarrow }\,\mathbf{vect}$ : $({v}*^{{\mathbf{v}}}{r})$ is the rescaling of a cotangent $v$ by the scalar $r$ .

We write $\overline{e} _{i}$ for program representing the $i$ -th canonical basis vector $e_i$ of type $\mathbf{vect}$ , and we write

(1.1) \begin{equation} \mathrm{Wrap}_{s}(x)\stackrel{\mathrm{def}}{=} \mathbf{case}\,{x}\,\mathbf{of}\,\langle{x_1,\ldots, x_s}\rangle\;{\rightarrow}\;{\langle \langle{x_1},{\overline{e} _{1}}\rangle, \ldots, \langle{x_s},{\overline{e} _{s}}\rangle \rangle }. \end{equation}

We define $[\![\mathbf{vect} ]\!] \stackrel{\mathrm{def}}{=} \mathbb{R}^\infty \stackrel{\mathrm{def}}{=}\sum _{k=0}^\infty \mathbb{R}^k$ as the infinite (vector space) coproduct of $k$ -dimensional real vector spaces. That is, we interpret $\mathbf{vect}$ as the type of dynamically sized real vectors.Footnote ³ We show that $\overleftarrow{\mathcal{D}}{(t)}$ implements the transposed derivative $D[\![t]\!] ^t$ of $[\![t]\!]$ in the following sense.

Theorem 2.3 (Reverse AD correctness, Theorem 9.1 with $k=\infty$ in main text). For any program $x:\tau \vdash{t}:\sigma$ for $\tau =\mathbf{real}^{s},\sigma =\mathbf{real}^l$ ,

\begin{align*} &[\![\mathbf{let}\,x ={\mathrm{Wrap}_{k}(x)}\,\mathbf{in}\,{\overleftarrow{\mathcal{D}}{(t)}}]\!] (x_1,\ldots, x_s)=\\ &\Big ((\pi _1([\![t]\!] (x_1,\ldots, x_s)), D[\![t]\!] ^t((x_{1},\ldots, x_s),e_1)),\ldots, (\pi _l([\![t]\!] (x_1,\ldots, x_s)), D[\![t]\!] ^t((x_{1},\ldots, x_s),e_l))\Big ) \end{align*}

for any $(x_1,\ldots, x_s)$ in the domain of definition of $[\![t]\!]$ .

We prove this theorem again using a similar logical relations argument, defining $T^{n}_{\tau }\subseteq (\mathbb{R}^n\to [\![\tau ]\!] )\times ((\mathbb{R}^n\times{(\mathbb{R}^\infty )}^n)\to [\![\overleftarrow{\mathcal{D}}{(\tau )}]\!] )$ and $P^{n}_{\tau }\subseteq (\mathbb{R}^n\rightharpoonup{[\![\tau ]\!] })\times (\mathbb{R}^n\times{(\mathbb{R}^\infty )}^n)\rightharpoonup{[\![\overleftarrow{\mathcal{D}}{(\tau )}]\!] })$ as before for all types $\tau$ of language, setting

\begin{align*} T^{n}_{\mathbf{real}}&\stackrel{\mathrm{def}}{=} \{(\gamma, \gamma ')\mid \gamma \text{ is differentiable and }\gamma '=(x, L)\mapsto (\gamma (x), L(D\gamma ^t(x, e_1)))\}\\ T^{n}_{\tau \,{\mathop{\times }}\,\sigma } & \stackrel{\mathrm{def}}{=} \{(x \mapsto (\gamma _1(x),\gamma _2(x)), (x,L)\mapsto (\gamma^{\prime}_1(x,L),\gamma^{\prime}_2(x,L)))\mid (\gamma _1,\gamma^{\prime}_1)\in T^{n}_{\tau }\text{ and } (\gamma _2,\gamma^{\prime}_2)\in T^{n}_{\sigma }\}\\ T^{n}_{\tau \,{\mathop{\sqcup }}\,+\sigma } & \stackrel{\mathrm{def}}{=} \{(\iota _1\circ \gamma _1,\iota _1\circ \gamma^{\prime}_1)\mid (\gamma _1,\gamma^{\prime}_1)\in T^{n}_{\tau }\}\cup \{(\iota _2\circ \gamma _2,\iota _2\circ \gamma^{\prime}_2)\mid (\gamma _2,\gamma^{\prime}_2)\in T^{n}_{\sigma }\}\\ T^{n}_{\tau \to \sigma }& \stackrel{\mathrm{def}}{=} \{(\gamma, \gamma ')\mid \forall (\delta, \delta ')\in T^{n}_{\tau }. (x\mapsto \gamma (x)(\delta (x)), (x,L)\mapsto \gamma '(x,L)(\delta '(x,L)))\in P^{n}_{\sigma }\}\\ T^n_{\boldsymbol{\mu }\alpha .\tau }&\stackrel{\mathrm{def}}{=} \{(\mathrm{roll}\circ \gamma, \mathrm{roll}\circ \gamma ')\mid (\gamma, \gamma ')\in T^n_{\tau{}[^{\boldsymbol{\mu }\alpha .\tau }\!/\!_{\alpha }]}\} \\ P^{n}_{\tau } & \stackrel{\mathrm{def}}{=} \Big \{(\gamma, \gamma ')\mid \gamma ^{-1}([\![\tau ]\!] )\times (\mathbb{R}^\infty ){}^n=\gamma ^{'-1}([\![\overleftarrow{\mathcal{D}}{(\tau )}]\!] )\text{ is open and for all differentiable}\\&\qquad \quad \!\!\! \delta :\mathbb{R}^n\,{\rightarrow }\, \gamma ^{-1}([\![\tau ]\!] ) \text{ we have } (\gamma \circ \delta, (x,L)\mapsto \gamma '(\delta (x),L\circ D\delta ^t(x,-))) \in T^{n}_{\tau }\Big \}, \end{align*}

where we consider $(\mathbb{R}^\infty ){}^n$ as a type of linear transformations from $\mathbb{R}^n$ to $\mathbb{R}^\infty$ and we write $e_i$ for the $i$ -th standard basis vector of $\mathbb{R}^n$ . We then prove the following “fundamental lemma,” using induction on the typing derivation of $t$ :

As $T^{s}_{\mathbf{real}^s}$ contains, in particular,

\begin{equation*}(\textrm {id}, ((x_1,\ldots, x_s), L)\mapsto ((x_1,L e_1),\ldots, (x_s,L e_s))),\end{equation*}

our theorem follows.

Extending to arrays

AD tends to be applied to programs that manipulate large arrays of reals. Seeing that such arrays are denotationally equivalent to lists $\boldsymbol{\mu }\alpha .\mathbf{1}\,{\mathop{\sqcup }}\,+\alpha \,{\mathop{\times }}\,\mathbf{real}$ , while only the computational complexity of operations differs, our correctness result also applies to functional languages with arrays. We thus differentiate array types ${\tau }[]$ with elements of type $\tau$ in the obvious structure preserving way, for example

\begin{align*} \mathcal{D}\;{({\tau }[])}\stackrel{\mathrm{def}}{=}{\mathcal{D}\;{(\tau )}}[] \qquad \mathcal{D}\;{(\mathbf{generate})}\stackrel{\mathrm{def}}{=} \mathbf{generate}\qquad \mathcal{D}\;{(\mathbf{map})}\stackrel{\mathrm{def}}{=}\mathbf{map}\qquad \mathcal{D}\;{(\mathbf{foldr})}\stackrel{\mathrm{def}}{=} \mathbf{foldr} \end{align*}

and similarly for dual numbers reverse AD.

Almost everywhere differentiability

Taking inspiration from Lee et al. (Reference Lee, Yu, Rival and Yang2020),Mazza and Pagani (Reference Mazza and Pagani2021), and Huot et al. (Reference Huot, Lew, Mansinghka and Staton2023), we can increase our ambitions and show that, given sufficiently nice primitive operations, our AD methods compute correct derivatives almost everywhere for (almost everywhere) terminating programs in our language. In fact, a minor adaptation of our methods yields these results. Indeed, as long as we assume that all our (partial) primitive operations denote functions that are piecewise analytic under analytic partition (PAP) and are defined on a $c$ -analytic subset (meaning: a countable union of analytic subsets) of $\mathbb{R}^n$ , then we can simply redefine our logical relations

\begin{align*} T^n_{\mathbf{real}} &\stackrel{\mathrm{def}}{=} \{(\gamma, \gamma ')\mid \gamma \text{ is PAP and } \gamma '=(x,v)\mapsto (\gamma (x),\gamma ''(x,v))\text{ for an intensional derivative $\gamma ''$ of $\gamma $} \}\\ &\{A_i\subseteq \mathbb{R}^n\}_{i\in I}\text{ of }\gamma ^{'-1}([\![\mathcal{D}\;{(\tau )}]\!] )\text{ and there exist open neighbourhoods $U_i$ of $A_i$ with functions }\\ &\gamma _i:U_i\,{\rightarrow }\, [\![\tau ]\!], \gamma^{\prime}_i:U_i\times \mathbb{R}^n\,{\rightarrow }\, [\![\mathcal{D}\;{(\tau )}]\!] \text{ such that }\gamma |_{A_i}=\gamma _i|_{A_i}\text{ and } \gamma '|_{A_i\times \mathbb{R}^n}=\gamma^{\prime}_i|_{A_i\times \mathbb{R}^n} \text{ and}\\ & \text{for all analytic }\delta :\mathbb{R}^n\,{\rightarrow }\, U_i\text{ we have that } (\gamma _i\circ \delta, (x,v)\mapsto (\gamma _i(\delta (x)), \gamma^{\prime}_i(D\delta (x,v))))\in T^n_{\tau }\Big \}. \end{align*}

to conclude that

• • any program $x:\tau \vdash{t}:\sigma$ for $\tau =\mathbf{real}^{m},\sigma =\mathbf{real}^l$ in our language denotes a partial PAP function $[\![{t}$ ]]defined on a $c$ -analytic subset;

• • our AD transformation $\mathcal{D}\;{(t)}$ computes $\left ( [\![t]\!], g\right )$ for an intensional derivative $g$ of $[\![t]\!]$ , which coincides almost everywhere in the domain with the (standard) derivative $D[\![t]\!]$ of $[\![t]\!]$ .

Consequently, if our program terminates almost everywhere, the AD transformation computes the correct derivative almost everywhere.

4.1 $CBV$ models: term recursion and iteration

In order to interpret our language defined in Section 6, we need an additional support for term recursion and iteration. Since we do not impose further equations for the iteration or recursion constructs in our language, the following definitions establish our class of models for term recursion and iteration. In contrast with most other references we are aware of, we give an explicit discussion of the case of iteration, even though it is definable in terms of recursion. The reason is that there are interesting PL with iteration but without recursion, of which we might want to prove properties.

It should be noted that $\mathfrak{C}_{\mathcal{BV}}$ has finite products. Given two $CBV$ models $\left ( \mathcal{V} _0, \mathcal{T} _ 0, \mu _0, {\mathsf{itt}} _0 \right )$ and $\left ( \mathcal{V} _1, \mathcal{T} _ 1, \mu _1, {\mathsf{itt}} _1 \right )$ , the product is given by

(10) \begin{equation} \left ( \mathcal{V} _0\times \mathcal{V} _1, \mathcal{T} _ 0\times \mathcal{T} _ 1, \left ( \mu _0, \mu _1\right ), \left ({\mathsf{itt}} _0,{\mathsf{itt}} _1\right ) \right ) \end{equation}

where $\left ( \mu _0 \times \mu _1\right )^{\left ( W, W'\right ), \left ( Y, Y' \right ) } = \mu _0 ^{W,Y}\times \mu _1 ^{W',Y'}$ and $\left ({\mathsf{itt}} _0 \times{\mathsf{itt}} _1\right )^{\left ( W, W'\right ), \left ( Y, Y' \right ) } = \left ({\mathsf{itt}} _0 ^{W,Y}\times{\mathsf{itt}} _1 ^{W',Y'} \right )$ .

5.1 Fixpoints: term recursion and iteration

Recall that, if $A$ is a pointed $\omega$ -cpo and $q \;:\; A\,{\rightarrow }\, A$ is an endomorphism in $\boldsymbol{\omega } \mathbf{Cpo}$ , then $q$ has a least fixed point given by the colimit of the chain

(12)

by Kleene’s fixpoint theorem. Given such an endomorphism, we denote by $\mathrm{lfp}\left ({q}\right )$ its least fixed point.

Henceforth, let $\left ( \mathcal{V}, \mathcal{T}\, \right )$ be a $CBV$ $\boldsymbol{\omega } \mathbf{Cpo}$ -pair and $J\;:\; \mathcal{V}\,{\rightarrow }\, \mathcal{C}$ the corresponding $\mathcal{V}$ -enriched universal Kleisli functor. We denote by $-\otimes - \;:\; \mathcal{V}\times \mathcal{C}\,{\rightarrow }\, \mathcal{C}$ the $\mathcal{V}$ -tensor product in $\mathcal{C}$ , also called the $\mathcal{V}$ -copower, which, in this case, corresponds to the usual action of $\mathcal{V}$ on $\mathcal{C}$ .

By hypothesis, for any $W,Y,Z\in \mathcal{V}$ , the $\omega$ -cpo $\mathcal{C} \left ( Z\otimes W, Y \right )\,{\cong}\, \mathcal{V}\left ( Z, \mathcal{C}\left [ W,Y \right ] \right )$ is pointed. Let us write $\Lambda _Z^{W,Y}$ for the isomorphism from left to right. Then, we can define

(13) \begin{eqnarray} \overline{\mu } ^{W,Y}_Z :&\mathcal{V}\left ({ Z\times \mathcal{C}\left [{W},{Y}\right ] },{\mathcal{C}\left [{W},{Y}\right ]}\right ) &\,{\rightarrow }\, \mathcal{V}\left ({ Z},{ \mathcal{C}\left [{W},{Y}\right ]}\right ) \nonumber \\ & f & \mapsto \mathrm{lfp}\left ({ h\mapsto f\circ \left ( Z\times h\right )\circ \partial _{Z} }\right ) \end{eqnarray}

(14) \begin{eqnarray}\qquad\qquad\qquad\qquad\qquad \overline{\mathsf{it}} ^{W,Y}_Z :&\mathcal{C}\left ({Z\otimes W},{ W\sqcup Y }\right ) &\,{\rightarrow }\, \mathcal{C}\left ({ Z\otimes W},{Y}\right ) \\ & f & \mapsto \mathrm{lfp}\left ({ \begin{array}{l} h\mapsto [ h, J\left ( \pi _{Y}\right )\! ]\circ d_{Z,W,Y} \circ \left ( Z\otimes f\right )\\ \qquad \quad \circ a_{Z,Z,W} \circ \left ( \partial _{Z} \otimes \mathrm{id} _ W \right ) \end{array} }\right ) \nonumber \end{eqnarray}

where $\partial _{Z} = \left (\mathrm{id}_ Z, \mathrm{id} _Z \right ) \;:\; Z\,{\rightarrow }\, Z\times Z$ is the diagonal morphism, $d_{Z,W,Y}:Z\otimes (W\sqcup Y)\,{\rightarrow }\, (Z\otimes W)\sqcup (Z\otimes Y)$ is the distributor, and $a_{Z,W,Y}:(Z\times W)\otimes Y\,{\rightarrow }\, Z\otimes (W\otimes Y)$ is the associator. Since the morphisms above are $\boldsymbol{\omega } \mathbf{Cpo}$ -natural in $Z\in \mathcal{V}$ , they give rise to the families of morphisms

(15) \begin{eqnarray} \mu _ \omega = \left ( \mu _ \omega ^{W,Y} \right ) _{(W,Y)\in \mathcal{C}\times \mathcal{C}} &\stackrel{\mathrm{def}}{=} & \left ( \overline{\mu } ^{W,Y} _{\mathcal{V}\left [{\mathcal{C}\left [{W},{Y}\right ] },{\mathcal{C}\left [{W},{Y}\right ]}\right ]}\left ( \mathsf{eval}_{ \mathcal{C}\left [{W},{Y}\right ], \mathcal{C}\left [{W},{Y}\right ] } \right ) \right ) _{(W,Y)\in \mathcal{C}\times \mathcal{C}} \end{eqnarray}

(16) \begin{align} \mathsf{it}_\omega = \left ( \mathsf{it}_\omega ^{W,Y} \right ) _{(W,Y)\in \mathcal{C}\times \mathcal{C} }\, &\stackrel{\mathrm{def}}{=} & \Lambda _{\mathcal{V}\left [{ W },{T\left ( W\sqcup Y\right ) }\right ]}^{W,Y}\left ( \overline{\mathsf{it}} ^{W,Y} _{\mathcal{V}\left [{ W },{T\left ( W\sqcup Y\right ) }\right ]} \left ( \mathsf{eval}_{ W, T\left (W\sqcup Y\right ) } \right )\right ) _{(W,Y)\in \mathcal{C}\times \mathcal{C}} \end{align}

by the Yoneda lemma, where $\mathsf{eval}_{ A, B } \;:\; \mathcal{V}\left [{A},{B}\right ] \times A\,{\rightarrow }\, B$ is the evaluation morphism given by the cartesian closed structure.

Proof Since $H$ is a $\boldsymbol{\omega } \mathbf{Cpo}$ -functor and, for any $\left ( W, Y\right )\in{\mathsf{ob}\,\mathcal{V}}\times{\mathsf{ob}\,\mathcal{V}}$ ,

\begin{equation*} H \;:\; \mathcal {V}\left ( {W},{TY}\right )\,{\rightarrow }\, \mathcal {V}\,'\left ( HW, T'HY \right ) \end{equation*}

is a pointed $\boldsymbol{\omega } \mathbf{Cpo}$ -morphism, we get that, indeed, $H$ respects the free iteration and free recursion as defined in (15) and (16).

It should be noted that, given $CBV$ $\boldsymbol{\omega } \mathbf{Cpo}$ -pairs $\left ( \mathcal{V} _0, \mathcal{T} _0 \right )$ and $\left ( \mathcal{V} _1, \mathcal{T} _1\right )$ ,

(17) \begin{equation} \left ( \mathcal{V} _0, \mathcal{T} _0\right )\times \left ( \mathcal{V} _1, \mathcal{T} _1 \right ) = \left ( \mathcal{V} _0\times \mathcal{V} _1, \mathcal{T} _0\times \mathcal{T} _1 \right ) \end{equation}

is the product in $\boldsymbol{\omega } \mathbf{CPO}\textrm{-}\mathfrak{C}_{\mathcal{BV}}$ . Moreover, $\mathcal{U}_{\mathcal{BV}}$ preserves finite products.

6.1 Source language as a standard call-by-value language with iteration and recursion

We consider a standard (coarse-grain) CBV language over a ground type $\mathbf{real}$ , certain real constants $\underline{c}\in \mathrm{Op}_0$ , certain primitive operations $\mathrm{op} \in \mathrm{Op}_n$ for each nonzero natural number $n\in \mathbb{N} ^\ast$ , and $\mathbf{sign}$ . We denote $\displaystyle \mathrm{Op} := \bigcup _{n\in \mathbb{N} } \mathrm{Op} _ n$ .

As it is clear from the semantics defined in Section 7.3, $\mathbf{real}$ intends to implement the real numbers. Moreover, for each $n\in \mathbb{N}$ , the operations in $\mathrm{Op} _n$ intend to implement partially defined functions $\mathbb{R} ^n \rightharpoonup \mathbb{R}$ . Finally, $\mathbf{sign}\,$ intends to implement the partially defined function $\mathbb{R}\rightharpoonup \mathbf{1}\sqcup \mathbf{1}$ defined on $\mathbb{R} ^- \cup \mathbb{R} ^+$ , which takes $\mathbb{R} ^-$ to the left component and $\mathbb{R} ^+$ to the right component.

Although it is straightforward to consider more general settings, we also add the assumption that the primitive operations implement differentiable functions (see Assumption 7.6).

We treat this operations in a schematic way as this reflects the reality of practical AD libraries, which are constantly being expanded with new primitive operations.

The types $\tau,\sigma,\rho$ , values ${v},{w},{u}$ , and computations ${t},{s},{r}$ of our language are as follows.

\begin{align*}\begin{array}{llllll} \tau, \sigma, \rho & ::= & \quad \text{types} &\mathrel{\lvert }\quad \, & \mathbf{1} \mathrel{\lvert } \tau _1 \,{\mathop{\times }}\, \tau _2 & \qquad \text{products}\\[-4pt] &\mathrel{\lvert } \quad\! \mathbf{real} & \qquad \text{numbers} &\mathrel{\lvert }& \tau \to \sigma & \qquad \text{function} \\[-4pt] &\mathrel{\lvert } \quad\! \mathbf{0} \mathrel{\lvert } \tau \,{\mathop{\sqcup }}\,+ \sigma & \qquad \text{sums}\\[8pt] {v},{w},{u} & ::= & \quad \text{values} &\mathrel{\lvert }\quad \, & \langle \,\rangle \ \mathrel{\lvert } \langle{{v}},{{w}}\rangle & \qquad \text{tuples}\\[-4pt] &\mathrel{\lvert }\quad\! x, y, z & \qquad \text{variables} &\mathrel{\lvert }& \lambda x.{t} & \qquad \text{abstractions}\\[-4pt] &\mathrel{\lvert }\quad\! \underline{c} & \qquad \text{constants}&\mathrel{\lvert } &\mu{x}.{t} & \qquad \text{term recursion}\\[-4pt] &\mathrel{\lvert }\quad\! \mathbf{inl}\,{{v}} \mathrel{\lvert } \mathbf{inr}\,{{v}} & \qquad \text{sum inclusions} \end{array}\end{align*}

\begin{align*}\begin{array}{llllll} {t},{s},{r} & ::= & \quad \text{computations} &\mathrel{\lvert }\quad \, & \langle \,\rangle \ \mathrel{\lvert } \langle{t},{s}\rangle & \qquad \text{tuples}\\[-4pt] &\mathrel{\lvert }\quad\! x, y, z & \qquad \text{variables}&\mathrel{\lvert }\quad \, & \mathbf{case}\,{s}\,\mathbf{of}\,\langle{x},{y}\rangle \to{t}\hspace{-10pt}\; & \qquad \text{product match}\\[-4pt]&\mathrel{\lvert } \quad\! \mathbf{let}\,{t} = x \,\mathbf{in}\,{s} & \qquad \text{sequencing} &\mathrel{\lvert }& \lambda x.{t} & \qquad \text{abstractions} \\[-4pt] &\mathrel{\lvert }\quad\! \underline{c} & \qquad \text{constant}&\mathrel{\lvert }&{t}\,\,{s} & \qquad \text{function app.} \\[-4pt] &\mathrel{\lvert } \quad\! \mathrm{op} ({t}_1,\ldots, {t}_n) & \qquad \text{operation}&\mathrel{\lvert }&\mathbf{iterate}\,{t}\,\mathbf{from}\,{x}={s}\hspace{-10pt}\; & \qquad \text{iteration}\\[-4pt]&\mathrel{\lvert } \quad\! \mathbf{case}\,{t}\,\mathbf{of}\{\,\} & \qquad \text{sum match}& \mathrel{\lvert } & \mu{x}.{t} & \qquad \text{term recursion}\\[-4pt]&\mathrel{\lvert }\quad\! \mathbf{inl}\,{t} \mathrel{\lvert } \mathbf{inr}\,{t} & \qquad \text{sum inclusions}&\mathrel{\lvert }&\mathbf{sign}\,{t} & \qquad \text{sign function}\\[-4pt] &\mathrel{\lvert } \quad\! \mathbf{case}\,{r}\,\mathbf{of}\ \{\begin{array}{l}\;\;\mathbf{inl}\, x\to{t}\\[-4pt] \mathrel{\lvert } \mathbf{inr}\, y \to{s}\end{array}\}\hspace{-15pt}\; & \qquad \text{sum match}\\[-4pt]\end{array}\end{align*}

We use sugar $\mathbf{if}\,r\,\mathbf{then}\,t\,\mathbf{else}\,s\, \stackrel{\mathrm{def}}{=} \mathbf{case}\,{\mathbf{sign}\,{r}}\,\mathbf{of}\;{\{\_\to{t}\mathrel{\big \lvert } \_\to{r}\}}$ , $\mathbf{fst}\,{t}\stackrel{\mathrm{def}}{=} \mathbf{case}\,{t}\,\mathbf{of}\,\langle{x},{\_}\rangle \to{x}$ , $\mathbf{snd}\,{t}\stackrel{\mathrm{def}}{=} \mathbf{case}\,{t}\,\mathbf{of}\,\langle{\_},{x}\rangle \to{x}$ , and $\mathbf{let} \,\mathbf{rec}\, f(x) = t\,\mathbf{in}\,s \stackrel{\mathrm{def}}{=} \mathbf{let}\,f = \mu f.\lambda x. t\,\mathbf{in}\,s$ . In fact, we can consider iteration as syntactic sugar as well:

$ \mathbf{iterate}\,{t}\,\mathbf{from}\,{x}={s}\stackrel{\mathrm{def}}{=}(\mu z.\lambda x.{ \mathbf{case}\,{t}\,\mathbf{of}\,\{\mathbf{inl}\, x'\to z\,x'\mid \mathbf{inr}\, x''\to x''\} })\,{s}$ .

The computations are typed according to the rules of Figs. 1 and 2, where $\mathtt{R}\subset \mathbb{R}$ is a fixed set of real numbers containing $0$ . For now, the reader may ignore the kinding contexts $\Delta$ . They will serve to support our treatment of ML-style polymorphism later.

Figure 1.

Typing rules for a basic source language with real conditionals, where $\mathtt{R}\subset \mathbb{R}$ is a fixed set of real numbers containing $0$ .

Figure 2.

Typing rules for term recursion and iteration.

We consider the standard CBV $\beta \eta$ -equational theory of Moggi (Reference Moggi1989) for our language, which we list in Fig. 3. We could impose further equations for the iteration construct as is done in Bloom and Ésik (Reference Bloom and Ésik1993) and Goncharov et al. (Reference Goncharov, Rauch and Schröder2015) as well as for the basic operations $\mathrm{op}$ and the sign function $\mathbf{sign}\,$ . However, such equations are unnecessary for our development.

Figure 3.

Basic $\beta \eta$ -equational theory for our language. We write $\beta \eta$ -equality as $\equiv$ to distinguish it from equality in let-bindings. We write ${\# x_1,\ldots, x_n}{\equiv }$ to indicate that the variables are fresh in the left-hand side. In the top right rule, $x$ may not be free in $r$ . Equations hold on pairs of computations of the same type.

6.2. Target language

We define our target language by extending the source language adding the following syntax, with the typing rules of Fig. 4.

\begin{align*} \begin{array}{lllllll}\tau, \sigma, \rho & ::= & & \quad \text{types}&\mathrel{\lvert }\quad \, & \mathbf{vect} & \qquad \text{(co)tangent}\\[-4pt] &\mathrel{\lvert }& \ldots & \qquad \text{as before}\\[10pt] {v},{w},{u} & ::= && \quad \text{values}&\mathrel{\lvert } & \overline{0} & \qquad \text{zero}\\[-4pt] &\mathrel{\lvert } &\overline{e} _{i}&\qquad \text{$i$-th canonical element} &\mathrel{\lvert } &{t} +{s}&\qquad \text{addition of vectors}\\[-4pt] &\mathrel{\lvert }&\ldots &\qquad \text{as before}&\mathrel{\lvert } &{t}\ast{s} & \qquad \text{scalar multiplication} \\[-4pt]&&&& \mathrel{\lvert } & \mathfrak{h} _{i} t & \qquad \text{proj. handler} \\[10pt]&& &&\mathrel{\lvert } & \overline{0} & \qquad \text{zero }\\[-4pt] {t},{s},{r} & ::= && \quad \text{computations}&\mathrel{\lvert } &{t} +{s}&\qquad \text{addition of vectors}\\[-4pt] &\mathrel{\lvert }&\ldots &\qquad \text{as before}&\mathrel{\lvert } &{t}\ast{s} & \qquad \text{scalar multiplication} \\[-4pt] &\mathrel{\lvert }&\overline{e} _{i}&\qquad \text{canonical element} & \mathrel{\lvert } & \mathfrak{h} _{i} t & \qquad \text{proj. handler} \end{array} \end{align*}

Figure 4.

Extra typing rules for the target language with iteration and recursion, where we denote $\mathbb{N} ^\ast := \mathbb{N} - \{ 0 \}$ , $\mathbf{real} ^1 := \mathbf{real}$ and $\mathbf{real} ^{i+1} = \mathbf{real} ^i \times \mathbf{real}$ .

The operational semantics of the target language depends on the intended behavior for the $AD$ macro $\mathcal{D}$ defined in Section 6.4. In our context, we want $\mathbf{vect}$ to implement a vector space ((co)tangent vectors), with the respective operations and the usual laws between the operations such as distributivity of the scalar multiplication over the vector addition (which is particularly useful for efficient implementations (Smeding and Vákár Reference Smeding and Vákár2023).

The terms $\mathfrak{h} _{i} t$ are irrelevant for the definition and correctness of the macro $\mathcal{D}$ , but it is particularly useful to illustrate the expected types in Section 9.6 and 9.7. Although this perspective is unimportant for our correctness statement, the reader might want to view $\mathbf{vect}$ as a computation type encompassing computational effects for the vector space operations $\overline{e}_i$ , $(\!*\!)$ , and $(+)$ with handlers given by the terms $\mathfrak{h} _{i} t$ .

We are particularly interested in the case that $\left ( \mathbf{vect}, +, \ast, \overline{0}\right )$ implements the vector space $\left ( \mathbb{R} ^k, +, \ast, 0\right )$ , for some $k\in \mathbb{N}\cup \left \{ \infty \right \}$ ,Footnote ⁷ where $\overline{e} _{i}$ implements the $i$ -th element $e^{k}_{i}\in \mathbb{R} ^k$ of the canonical basis if $k=\infty$ or if $i\leq k$ , and $0\in \mathbb{R}^k$ otherwise. In this case, $\mathfrak{h} _{i} t$ is supposed to implement

(18) \begin{equation} \mathfrak{p}_{k \rightarrow i}\;:\; \mathbb{R} ^k\,{\rightarrow }\, \mathbb{R} ^i, \end{equation}

which denotes the canonical projection if $i\leq k$ and the coprojection otherwise.

For short, we say that $\mathbf{vect}$ implements the vector space $\mathbb{R} ^k$ to refer to the case above. It corresponds to the $k$ -semantics for the target language defined in Section 7.4.

Figure 5.

Assignment that gives the universal property of the source language.

Figure 6.

Assignment that gives the universal property of the target language.

6.3 The $CBV$ models $( \mathbf{Syn}_V, \mathbf{Syn}_{\mathcal{S}},\mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} )$ and $( \mathbf{Syn}_V^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathcal{S}}^{{\mathbf{tr}}}, \mathbf{Syn}_\mu ^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathsf{it}}^{{\mathbf{tr}}} )$

As discussed in Appendix A, we can translate our coarse-grain languages to fine-grain CBV languages. The fine-grain languages corresponding to the source and target languages correspond to the $CBV$ models

(19) \begin{equation} \left ( \mathbf{Syn}_V, \mathbf{Syn}_{\mathcal{S}},\mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} \right )\qquad \mbox{and}\qquad \left ( \mathbf{Syn}_V^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathcal{S}}^{{\mathbf{tr}}}, \mathbf{Syn}_\mu ^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathsf{it}}^{{\mathbf{tr}}} \right ) \end{equation}

with the following universal properties.

6.4. Dual numbers AD transformation for term recursion and iteration

Let us fix, for all $n\in \mathbb{N}$ , for all $\mathrm{op} \in \mathrm{Op}_n$ , and for all $1\leq i \leq n$ , computations $x_1:\mathbf{real},\ldots, x_n:\mathbf{real}\vdash \partial _i\mathrm{op} (x_1,\ldots, x_n):\mathbf{real}$ , which represent the partial derivatives of $\mathrm{op}$ . Using these terms for representing partial derivatives, we define, in Fig. 7, a structure preserving macro $\mathcal{D}$ on the types and computations of our language for performing AD.

We extend $\mathcal{D}$ to contexts: $\mathcal{D}\;{(\{x_1\;{:}\;\tau _1,{.}{.}{.},x_n{:}\tau _n\})}\stackrel{\mathrm{def}}{=} \{x_1\;{:}\;\mathcal{D}\;{(\tau _1)},{.}{.}{.},x_n\;{:}\;\mathcal{D}\;{(\tau _n)}\}$ . This turns $\mathcal{D}$ into a well-typed, functorial macro in the following sense.

Our macro $\mathcal{D}$ can be seen as a class of macros, since it depends on the target language. More precisely, it depends on what $\mathbf{vect}$ implements (see Section 6.2).

Figure 7.

AD macro $\mathcal{D}\;{(-)}$ defined on types and computations. All newly introduced variables are chosen to be fresh. We provide a more efficient way of differentiating $\mathbf{sign}\,$ in Appendix B.

As an example, for the program of Equation (1), $\mathcal{D}$ computes, modulo some $\beta \eta$ -equality to aid legibility, the following derivative (where we also define $\mathcal{D}\;{(\mathbf{int})}\stackrel{\mathrm{def}}{=}\mathbf{int}$ , $\mathcal{D}\;{({t}\lt{s}\lt{r})}\stackrel{\mathrm{def}}{=} \mathbf{fst}\,(\mathcal{D}\;{(t)})\lt \mathbf{fst}\,(\mathcal{D}\;{(s)})\lt \mathbf{fst}\,(\mathcal{D}\;{(r)})$ , and $\partial _i(+)(x,y)\stackrel{\mathrm{def}}{=} 1$ ):

\begin{align*} \mathbf{iterate}\,\Big (\begin{array}{l} \mathbf{case}\,{z}\,\mathbf{of}\,\langle{i},{\langle{y'_{\!\!1}},{y'_{\!\!2}}\rangle }\rangle \to \mathbf{let}\,\langle y_1, y_2\rangle ={\mathcal{D}(t)(i, x)}\,\mathbf{in}\\ \mathbf{case}\,{-\epsilon \lt y_1 \lt \epsilon }\,\mathbf{of}\,\{\mathbf{inl}\,\_ \to \mathbf{inr}\, x\mid \mathbf{inr}\,\_\to \mathbf{inl}\, \langle{i + 1},{\langle{y_1 +y'_{\!\!1}},{y_2+y'_{\!\!2}}\rangle }\rangle \}\end{array}\Big )\\ \mathbf{from}\,{z}={\langle{0},{0}\rangle }. \end{align*}

6.5. AD transformation as a $CBV$ model morphism

By the universal property of $\left ( \mathbf{Syn}_V, \mathbf{Syn}_{\mathcal{S}},\mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} \right )$ established in Proposition 6.1, the assignment defined in Fig. 8 induces a unique $CBV$ model morphism

(20) \begin{equation} \mathbb{D} \;:\; \left ( \mathbf{Syn}_V, \mathbf{Syn}_{\mathcal{S}},\mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} \right )\,{\rightarrow }\, \left ( \mathbf{Syn}_V^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathcal{S}}^{{\mathbf{tr}}},\mathbf{Syn}_\mu ^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathsf{it}}^{{\mathbf{tr}}} \right ). \end{equation}

Figure 8.

AD assignment.

The macro $\mathcal{D}$ defined in Fig. 7 is encompassed by (20).

7.1 Basic concrete model

The most fundamental example of a $CBV$ $\boldsymbol{\omega } \mathbf{Cpo}$ -pair is given by $\left ( \boldsymbol{\omega } \mathbf{Cpo}, \left ( -\right )_\bot \right )$ where $\left ( -\right )_\bot$ is the (lax idempotent) monad that freely adds a least element $\bot$ to each $\omega$ -cpo. Indeed, of course, $\boldsymbol{\omega } \mathbf{Cpo}\left ( W, \left ( Y\right )_\bot \right )$ is pointed for any pair $\left ( W,Y\right )\in{\mathsf{ob}\,\boldsymbol{\omega } \mathbf{Cpo}}\times{\mathsf{ob}\,\boldsymbol{\omega } \mathbf{Cpo}}$ .

We consider the product $\left ( \boldsymbol{\omega } \mathbf{Cpo}, \left ( -\right )_\bot \right )\times \left ( \boldsymbol{\omega } \mathbf{Cpo}, \left ( -\right )_\bot \right ) = \left ( \boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}, \left ( -\right )_\bot \right )$ , where, by abuse of language, $\left ( (C, C^{\prime})\right )_\bot = \left ( \left (C\right )_\bot, \left (C^{\prime}\right )_\bot \right )$ . By Lemma 5.2, we obtain $CBV$ models

\begin{align*} &\mathcal{U}_{\mathcal{BV}} \left ( \boldsymbol{\omega } \mathbf{Cpo}, \left (-\right )_\bot \right )\qquad \text{and}\\ &\mathcal{U}_{\mathcal{BV}} \left ( \boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}, \left (-\right )_\bot \right ) = \mathcal{U}_{\mathcal{BV}} \left ( \boldsymbol{\omega } \mathbf{Cpo}, \left (-\right )_\bot \right )\times \mathcal{U}_{\mathcal{BV}} \left ( \boldsymbol{\omega } \mathbf{Cpo}, \left (-\right )_\bot \right ). \end{align*}

For example, the program from Equation (1) is interpreted as the function

(21) \begin{align} \mathbb{R} &\,{\rightarrow }\,{\mathbb{R}}_{\bot } \end{align}

(22) \begin{align} x&\mapsto \left \{ \begin{array}{ll} \bot & \text{if }N_{[\![t]\!], x}=\infty \\ \bot & \text{if }[\![t]\!] (i,x)=\epsilon \text{ for some }i\lt N_{[\![t]\!], x}\\ \sum _{i=0}^{N_{[\![t]\!], x}-1}[\![t]\!] (i,x) & \text{otherwise} \end{array}\right . \end{align}

where

• • $\lceil z\rceil ^\epsilon = z$ if $|z|\gt \epsilon$ , $\lceil z\rceil ^\epsilon = 0$ if $|z|\lt \epsilon$ and $\lceil z\rceil ^\epsilon = \bot$ otherwise;

• • $N_{[\![t]\!], x}$ is the smallest natural number $i$ such that $\lceil [\![t]\!] (i,x)\rceil ^\epsilon =0$ .

7.2 Differentiable functions and interleaved derivatives

Henceforth, unless stated otherwise, the cartesian spaces $\mathbb{R} ^n$ and its subspaces are endowed with the respective discrete $\boldsymbol{\omega } \mathbf{Cpo}$ -structures, in which $r\leq r'$ if and only if $r=r'$ .

Definition 7.1 (Interleaving function). For each $(n,k)\in \mathbb{N} \times \left ( \mathbb{N}\cup \left \{ \infty \right \}\right )$ , denoting by $\mathbb{I}_{n}$ the set $\left \{ 1,\ldots, n\right \}$ , we define the isomorphism (in $\boldsymbol{\omega } \mathbf{Cpo}$ with the respective discrete $\boldsymbol{\omega } \mathbf{Cpo}$ -structures)

(23) \begin{eqnarray} {\phi} _{n, k} \;:\; & \mathbb{R} ^n\times \left ( \mathbb{R} ^k\right ) ^n &\,{\rightarrow }\, \left ( \mathbb{R} \times \mathbb{R} ^k \right ) ^{n}\\ &\left ( (x_j)_{j\in \mathbb{I}_{n}}, (y_j)_{j\in \mathbb{I}_{n}} \right ) & \mapsto \left ( x_j, y_j \right ) _{j\in \mathbb{I}_{n}}.\nonumber \end{eqnarray}

For each open subset $U\subset \mathbb{R}^n$ , we denote by ${\phi} _{n, k}^U \;:\; U\times \left ( \mathbb{R} ^k\right ) ^n\,{\rightarrow }\, {\phi} _{n,k}\left ( U\times \left ( \mathbb{R} ^k\right ) ^n \right )$ the isomorphism obtained from restricting ${\phi} _{n,k}$ .

In Definition 7.2, Remark 7.3, and Lemma 7.4, let $\displaystyle g \;:\; U\,{\rightarrow }\, \coprod _{j\in L} V_j$ be a map where $U$ is an open subset of $\mathbb{R} ^n$ , and, for each $i\in L$ , $V_i$ is an open subset of $\mathbb{R} ^{m_i}$ .

Definition 7.2 (Derivative). The map $g$ is differentiable if, for any $i\in L$ , $g^{-1}\left ( V_i \right ) =W _i$ is open in $\mathbb{R} ^n$ and the restriction $g|_{W_i} \;:\; W_ i\,{\rightarrow }\, V_i$ is differentiable w.r.t the submanifold structures $W_i\subset \mathbb{R} ^{n}$ and $ V_i \subset \mathbb{R} ^{m_i}$ . In this case, for each $k\in \left ( \mathbb{N}\cup \left \{ \infty \right \}\right )$ , we define the function:

(24) \begin{eqnarray} \mathfrak{D}^{k}{g} \;:\; & {\phi} _{n, k}\left ( U\times \left ( \mathbb{R} ^k\right ) ^n\right ) &\,{\rightarrow }\, \coprod _{j\in L}\left ( {\phi} _{{m_j}, k}\left ( V_i\times \left ( \mathbb{R} ^k\right ) ^{m_i}\right )\right ) \\ & z &\mapsto \iota _{m_j}\circ {\phi} _{{m_j},k}^{V_j}\left ( g(x), \tilde{w}\cdot g'(x)^t\right ), \text{if } {\phi} _{n, k} ^{-1} \left ( z \right ) =\left ( x, w \right )\in W_i\times \left (\mathbb{R} ^k\right ) ^n \nonumber \end{eqnarray}

in which $\tilde{w}$ is the linear transformation $ \mathbb{R} ^n\,{\rightarrow }\, \mathbb{R} ^k$ corresponding to the vector $w$ , $\cdot$ is the composition of linear transformations, $\iota _{m_i}$ is the obvious $ith$ -coprojection of the coproduct (in the category $\boldsymbol{\omega } \mathbf{Cpo}$ ), and $g'(x)^t:\mathbb{R}^{m_i}\,{\rightarrow }\,\mathbb{R}^n$ is the transpose of the derivative $g'(x) :\mathbb{R} ^n\,{\rightarrow }\,\mathbb{R} ^{m_i}$ of $g|_{W_i} \;:\; W_ i\,{\rightarrow }\, V_i$ at $x\in U$ .

For example, the partial function $h$ of Equation (21) has the following derivative $\mathfrak{D}^{k}{(} h)$ :

(28) \begin{align} \mathbb{R}\times \mathbb{R}^k &\,{\rightarrow }\,{(\mathbb{R}\times \mathbb{R}^k)}_{\bot } \end{align}

(29) \begin{align} (x,v)&\mapsto \left \{ \begin{array}{ll} \bot & \text{if }N_{[\![t]\!], x}=\infty \\ \bot & \text{if }[\![t]\!] (i,x)=\epsilon \text{ for some }i\lt N_{[\![t]\!], x}\\ \sum _{i=0}^{N_{[\![t]\!], x}-1}\mathfrak{D}^{k}{(} [\![t]\!] (i,-))(x,v) & \text{otherwise} \end{array}\right . \end{align}

7.3 The semantics for the source language

We give a concrete semantics for our language, interpreting it in the $CBV$ $\boldsymbol{\omega } \mathbf{Cpo}$ -pair $\left ( \boldsymbol{\omega } \mathbf{Cpo}, \left (-\right )_\bot \right )$ .

We denote by $\mathbb{R}$ the discrete $\omega$ -cpo of real numbers, in which $r\leq r'$ if and only if $r=r'$ , and we define $ \mathsf{sign} \;:\; \mathbb{R}\,{\rightarrow }\, \left ( \mathsf{1} \sqcup \mathsf{1} \right )_\bot$ by (31), where $\iota _{1}, \iota _{2} \;:\; \mathsf{1}\,{\rightarrow }\, \mathsf{1}\sqcup \mathsf{1}$ are the two coprojections of the coproduct.

(30) \begin{equation} [\![-]\!] \;:\; \left ( \mathbf{Syn}_V, \mathbf{Syn}_T, \mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} \right ) \,{\rightarrow}\, \mathcal{U}_{\mathcal{BV}} \left ( \boldsymbol{\omega } \mathbf{Cpo}, \left ( -\right )_\bot \right ) \end{equation}

(31) \begin{equation} \mathsf{sign} (x) = \begin{cases} \bot, & \text{if } x = 0 \\ \iota _{1} (\ast ), & \text{if } x\lt 0 \\ \iota _{2} (\ast ), & \text{if } x\gt 0 \end{cases} \end{equation}

By the universal property of $\left ( \mathbf{Syn}_V, \mathbf{Syn}_{\mathcal{S}},\mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} \right )$ , there is only one $CBV$ model morphism (30) consistent with the assignment of Fig. 9 where $\mathsf{c}$ is the constant that $\underline{c}$ intends to implement, and, for each $\mathrm{op} \in \mathrm{Op} _n$ , ${f}_{\mathrm{op}}$ is the partial map that $\mathrm{op}$ intends to implement.

Figure 9.

Semantics’ assignment for each primitive operation $\mathrm{op} \in \mathrm{Op}_n$ ( $n\in \mathbb{N}$ ) and each constant $c\in \mathtt{R}$ .

The $CBV$ model morphism (30) (or, more precisely, the underlying functor of the $CBV$ morphism $[\![-]\!]$ ) gives the semantics for the source language. Although our work holds for more general contexts, we consider the following assumption over the semantics of our language.

7.4 The $k$ -semantics for the target language

For each $k\in \mathbb{N} \cup \left \{ \infty \right \}$ , we define the $k$ -semantics for the target language by interpreting $\mathbf{vect}$ as the vector space $\mathbb{R}^k$ . Namely, we extend the semantics $[\![-]\!]$ of the source language into a $k$ -semantics of the target language. More precisely, by Proposition 6.1, there is a unique $CBV$ model morphism (32) that extends $[\![-]\!]$ and is consistent with the assignment given by the vector structure (33) together with the projection (coprojection) $[\![\mathfrak{h} _{i} ]\!] _{k} \;:\; \mathbb{R} ^k\,{\rightarrow }\, \mathbb{R} ^i$ if $i\leq k$ ( $i\geq k$ ), for each $i\in \mathbb{N} ^\ast$ .

(32) \begin{eqnarray} [\![-]\!] _{k} \;:\; \left ( \mathbf{Syn}_V^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathcal{S}}^{{\mathbf{tr}}},\mathbf{Syn}_\mu ^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathsf{it}}^{{\mathbf{tr}}} \right ) &\,{\rightarrow }\, &\mathcal{U}_{\mathcal{BV}} \left ( \boldsymbol{\omega } \mathbf{Cpo}, \left (- \right )_\bot \right ) \end{eqnarray}

(33) \begin{eqnarray} \left ( [\![\mathbf{vect} ]\!] _{k}, [\![ + ]\!] _{k}, [\![ \ast ]\!] _{k}, [\![\overline{0} ]\!] _{k} \right ) &:= & \left ( \mathbb{R} ^k, +, \ast, 0 \right ) \end{eqnarray}

7.5 Soundness of $\mathcal{D}$ for primitive operations

For each $j\in \mathbb{I}_{n}$ , given a differentiable function $f \;:\; \mathbb{R} ^n\,{\rightarrow }\, \left ( \mathbb{R}\right )_\bot$ , we denote by $\mathfrak{d}_{j}\left ( f \right ) \;:\; \mathbb{R} ^n\,{\rightarrow }\, \left ( \mathbb{R}\times \mathbb{R}\right )_\bot$ the function defined by $\mathfrak{d}_{j}\left ( f \right ) \left ( x_1,\ldots, x_n\right ) = \mathfrak{d}^{1}\left ( f \right ) \circ {\phi} _{n,1} \left ( (x_1,\ldots, x_n), e^{n}_{j}\right )$ , where $e^{n}_{j}$ the $j$ -th vector of the canonical basis of $\mathbb{R}^n$ .

Lemma 7.8. The macro $\mathcal{D}$ defined in Fig. 7 is sound for primitives provided that

(34) \begin{equation} [\![\langle{\mathrm{op} (y_1,\ldots, y_n)},{\partial _j\mathrm{op} (y_1,\ldots, y_n) }\rangle ]\!] = \mathfrak{d}_{j}\left ( [\![\mathrm{op} ]\!] \right ), \end{equation}

for any primitive operation $\mathrm{op} \in \mathrm{Op} _ n$ of the source language.

8.1 Scone: proof-relevant categorical logical relations

Given an $\boldsymbol{\omega } \mathbf{Cpo}$ -functor $G:\mathcal{B}\,{\rightarrow }\,\mathcal{D}$ , the comma $\boldsymbol{\omega } \mathbf{Cpo}$ -category $\mathcal{D}\downarrow G$ of the identity along $G$ in ${\boldsymbol{\omega } \mathbf{Cpo}}\textrm{-}\mathbf{Cat}$ is defined as follows.

1. – The objects of $\mathcal{D}\downarrow G$ are triples $(D\in \mathcal{D}, C\in \mathcal{B}, j:D\,{\rightarrow }\, G(C) )$ in which $j$ is a morphism of $\mathcal{D}$ ; we think of these as pairs of a $\mathcal{B}$ -object $C$ and a proof-relevant predicate $(D, j)$ on $G(C)$ ;

2. – a morphism $(D,C, j)\,{\rightarrow }\, (D', C^{\prime}, h)$ between objects of $\mathcal{D}\downarrow G$ is a pair (35) making (36) commutative in $\mathcal{D}$ ; we think of these as $\mathcal{B}$ -morphisms $\alpha _1$ that respect the predicates, as evidenced by $\alpha _0$ :

(35) \begin{equation} \alpha = \left ( \alpha _0 \;:\; D\,{\rightarrow }\, D', \alpha _1 \;:\; C\,{\rightarrow }\, C^{\prime}\right ) \end{equation}

(36)

1. – if $\alpha = \left ( \alpha _0 \;:\; D\,{\rightarrow }\, D', \alpha _1 \;:\; C\,{\rightarrow }\, C^{\prime}\right ), \beta = \left ( \beta _0 \;:\; D\,{\rightarrow }\, D', \beta _1 \;:\; C\,{\rightarrow }\, C^{\prime}\right ) \;:\; \left ( D, C, j\right )\,{\rightarrow }$ $\left ( D', C^{\prime}, h\right )$ , are two morphisms of $\mathcal{D}\downarrow G$ , we have that $\alpha \leq \beta$ if $\alpha _0\leq \beta _0$ in $\mathcal{D}$ and $\alpha _1\leq \beta _1$ in  $\mathcal{B}$ .

Following the approach of Lucatelli Nunes and Vákár (Lucatelli Nunes Reference Lucatelli Nunes2022, Section 9), we have:

Theorem 8.1 (Monadic-comonadic scone). Let $G\;:\; \mathcal{B}\,{\rightarrow }\,\mathcal{D}$ be a right $\boldsymbol{\omega } \mathbf{Cpo}$ -adjoint functor. Assuming that $\mathcal{D}$ has finite $\boldsymbol{\omega } \mathbf{Cpo}$ -products and $\mathcal{B}$ has finite $\boldsymbol{\omega } \mathbf{Cpo}$ -coproducts, the $\boldsymbol{\omega } \mathbf{Cpo}$ -functor

(37) \begin{equation} \mathcal{L} \;:\; \mathcal{D} \downarrow G\,{\rightarrow }\, \mathcal{D}\times \mathcal{B}, \end{equation}

defined by $\left ( D\in \mathcal{D}, C\in \mathcal{B}, j:D\,{\rightarrow }\, G(C) \right )\mapsto \left ( D, C \right )$ , is $\boldsymbol{\omega } \mathbf{Cpo}$ -comonadic and $\boldsymbol{\omega } \mathbf{Cpo}$ -monadic. Footnote ⁹ This implies, in particular, that $\mathcal{L}$ creates (and strictly preserves) $\boldsymbol{\omega } \mathbf{Cpo}$ -limits and colimits. Footnote ¹⁰

By Theorem 8.1 and the enriched adjoint triangle theorem,Footnote ¹¹ we have:

Theorem 8.1 and Corollary 8.2 are $\boldsymbol{\omega } \mathbf{Cpo}$ -enriched versions of the fundamental results of Lucatelli Nunes and Vákár (Lucatelli Nunes Reference Lucatelli Nunes2022, Section 9). The details and proofs are presented in Appendix C.

8.2 Subscone: proof-irrelevant categorical logical relations

Henceforth, we assume that $\mathbf{Sub}\left ( \mathcal{D}\downarrow G \right )$ is a fullFootnote ¹² reflectiveFootnote ¹³ and repleteFootnote ¹⁴ $\boldsymbol{\omega } \mathbf{Cpo}$ -subcategory of $\mathcal{D}\downarrow G$ . We denote, herein, by $\mathfrak{T}_{sub}$ the idempotent $\boldsymbol{\omega } \mathbf{Cpo}$ -monad induced by the $\boldsymbol{\omega } \mathbf{Cpo}$ -adjunction.

Recall that a morphism $q$ in $\boldsymbol{\omega } \mathbf{Cpo}$ is full if its underlying functor is full. In this case, the underlying functor is also faithful and injective on objects. Moreover, a morphism $j$ in an $\boldsymbol{\omega } \mathbf{Cpo}$ -category $\mathcal{B}$ is full if $\mathcal{B}\left ( B, j \right )$ is full in $\boldsymbol{\omega } \mathbf{Cpo}$ for any $B\in \mathcal{B}$ .

Furthermore, recall that an $\boldsymbol{\omega } \mathbf{Cpo}$ -functor $H:\mathcal{W}\,{\rightarrow }\,\mathcal{Z}$ is locally full if, for any $\left ( X,W \right )\in{\mathsf{ob}\,\mathcal{W}}\times{\mathsf{ob}\,\mathcal{W}}$ , the morphism $H \;:\; \mathcal{W}\left ( X, W \right )\,{\rightarrow }\, \mathcal{Z}\left ( HX, HW \right )$ is a full $\boldsymbol{\omega } \mathbf{Cpo}$ -morphism. It should be noted that the $2$ -functor underlying a locally full $\boldsymbol{\omega } \mathbf{Cpo}$ -functor is locally fully faithful. Moreover, since every full morphism in $\boldsymbol{\omega } \mathbf{Cpo}$ is injective on objects, every locally full $\boldsymbol{\omega } \mathbf{Cpo}$ -functor is faithful (locally injective on objects).

(38)

(39)

We denote by $\underline{\mathcal{L}} \;:\; \mathbf{Sub}\left ( \mathcal{D}\downarrow G \right )\,{\rightarrow }\, \mathcal{B}$ the $\boldsymbol{\omega } \mathbf{Cpo}$ -functor given by the composition (38) where the unlabeled arrow is the full inclusion.

9.1 Fixing a particular subscone $ \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$

Henceforth, we follow the notation and definitions established in Section 7. In particular, unless stated otherwise, the cartesian spaces $\mathbb{R} ^n$ and its subspaces are endowed with the discrete $\boldsymbol{\omega } \mathbf{Cpo}$ -structure, in which $r\leq r'$ if and only if $r=r'$ .

For each $\left ( n, k\right ) \in \mathbb{N}\times \left ( \mathbb{N}\cup \left \{ \infty \right \} \right )$ , we define the $\boldsymbol{\omega } \mathbf{Cpo}$ -functor (41). We consider the full reflective $\boldsymbol{\omega } \mathbf{Cpo}$ -subcategory $ \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ of $\boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k}$ whose objects are triples (42) such that $j$ is full (and, hence, injective on objects).

(41) \begin{equation} G_{n, k} \stackrel{\mathrm{def}}{=} \boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}\left ( \left ( \mathbb{R}^n, \left ( \mathbb{R}\times \mathbb{R}^k\right ) ^n \right ), \left ( -, - \right ) \right ) \;:\; \boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}\,{\rightarrow }\, \boldsymbol{\omega } \mathbf{Cpo} \end{equation}

(42) \begin{equation} \left ( D\in \boldsymbol{\omega } \mathbf{Cpo}, \, \left ( C, C^{\prime}\right )\in \boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo},\, \left ( j\;:\; D\,{\rightarrow }\, G_{n, k}\left ( C, C^{\prime} \right )\right )\in \boldsymbol{\omega } \mathbf{Cpo} \right ) \end{equation}

That is, we are considering what Barthe et al. (2020) calls open logical relations (where closed logical relations would correspond to the case of $G_{n, k}=\boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}\left ( (1,1), - \right )$ ).

The $\boldsymbol{\omega } \mathbf{Cpo}$ -functor $G_{n, k}$ together with $ \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ satisfies Assumption 8.3. Therefore:

9.2 Lifting the partiality monad to the subscone

Let $(n,k)\in \mathbb{N}\times \left ( \mathbb{N}\cup \left \{\infty \right \}\right )$ . In order to get a categorical model of our language, we need to define a partiality monad for $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ .

We denote by $\mathfrak{O}_{n}$ the set of proper open non-empty subsets of the cartesian space $\mathbb{R}^n$ . For each $U\in \mathfrak{O}_{n}$ , we define

\begin{eqnarray*} \mathsf{Diff}_{\left (U, n, k\right )} &\stackrel{\mathrm{def}}{=} &\left ( \left \{ \left ( g\;:\; \mathbb{R} ^n\,{\rightarrow }\, U, \mathfrak{D}^{k}{g} \right ) \;:\; g \mbox{ is differentiable} \right \}, \left ( U, {\phi} _{n,k}\left ( U\times \left ( \mathbb{R} ^k\right ) ^n\right ) \right ), \mathrm{incl.}\right ) \\ &\in &{\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )} . \end{eqnarray*}

We define the $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )$ -monad $\mathcal{P}_{n, k}\left (-\right )_\bot$ on $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )$ by

(43) \begin{equation} \mathcal{P}_{n, k}\left (D, \left ( C, C^{\prime}\right ), j\right )_\bot \stackrel{\mathrm{def}}{=} \left ( \underline{\mathcal{P}_{n, k}\left (D, \left ( C, C^{\prime}\right ), j\right )_\bot }, \left ( \left ( C\right )_\bot, \left (C^{\prime} \right )_\bot \right ), \mathtt{j}_{X} \right ) \end{equation}

where $\underline{\mathcal{P}_{n, k}\left (D, \left ( C, C^{\prime}\right ), j\right )_\bot }$ is the union

(44) \begin{equation} \left \{ \bot \right \} \sqcup D \sqcup \left ( \coprod _{U\in \mathfrak{O}_{n} } \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )\left ( \mathsf{Diff}_{\left (U, n, k\right )}, \left ( D, \left ( C, C^{\prime}\right ), j\right ) \right ) \right ) \end{equation}

with the full $\boldsymbol{\omega } \mathbf{Cpo}$ -substructure of $ G_{n, k} \left ( \left ( C\right )_\bot, \left ( C^{\prime}\right )_\bot \right )$ induced by the inclusion $\mathtt{j}_{X}$ which is defined by the following components:

• • the inclusion $\left \{ \bot \right \}\,{\rightarrow }\,G_{n, k} \left ( \left ( C\right )_\bot, \left ( C^{\prime}\right )_\bot \right )$ of the least morphism $\bot \;:\; \left ( \mathbb{R} ^n, \left ( \mathbb{R} \times \mathbb{R} ^k\right ) ^n \right )\,{\rightarrow }\, \left ( \left ( C\right )_\bot, \left ( C^{\prime}\right )_\bot \right )$ in $\boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}\left ( \left ( \mathbb{R} ^n, \left ( \mathbb{R} \times \mathbb{R} ^k \right ) ^n \right ), \left ( \left ( C\right )_\bot, \left ( C^{\prime}\right )_\bot \right ) \right )$ ;

• • the inclusion of the total functions $G_{n, k}\left ( \eta _{C}, \eta _{C^{\prime}} \right )\circ j \;:\; D\,{\rightarrow }\, G_{n, k} \left ( C, C^{\prime} \right )\,{\rightarrow }\, G_{n, k} \left ( \left ( C\right )_\bot, \left ( C^{\prime}\right )_\bot \right )$ ;

• • the injections $\displaystyle \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )\left ( \mathsf{Diff}_{\left (U, n, k\right )}, \left ( D, \left ( C, C^{\prime}\right ), j\right ) \right )\,{\rightarrow }\, G_{n, k} \left ( \left ( C\right )_\bot, \left ( C^{\prime}\right )_\bot \right )$ , for $U\in \mathfrak{O}_{n}$ , defined by

  \begin{align*} \Big( \alpha _0, \alpha _1 & \left. = \left ( \beta _0 \;:\; U\,{\rightarrow }\, C, \beta _1 \;:\; {\phi} _{n, k}\left ( U\times \left ( \mathbb {R} ^k \right ) ^n\right ) \,{\rightarrow }\, C^{\prime} \right ) \right )\\& \mapsto \left( \overline {\beta _0} \;:\; \mathbb {R} ^n\,{\rightarrow }\, \left ( C\right )_\bot, \overline {\beta _1} :\left ( \mathbb {R} \times \mathbb {R} ^k\right ) ^n\,{\rightarrow }\, \left ( C^{\prime}\right )_\bot \right ), \end{align*}
  where $\overline{\beta _0}$ and $\overline{\beta _1}$ are the respective corresponding canonical extensions. The image of $\mathtt{j}_{X}$ forms a sub- $\omega$ -cpo because the union $\bigcup _{n\in \mathbb{N}} U_n$ of open sets $U_n$ is open and because $D$ is an $\omega$ -cpo.

For each $(C,C^{\prime})\in \boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}$ , the component $\left ( \mathrm{m} _{C}, \mathrm{m} _{C^{\prime}} \right )$ and $\left ( \eta _{C}, \eta _{C^{\prime}} \right )$ of the multiplication and the unit of the monad $\left ( -\right )_\bot$ on $\boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}$ define morphisms

(45) \begin{eqnarray} \overline{\mathrm{m}} _{\left ( D, \left ( C, C^{\prime}\right ), j \right ) } \;:\; &\mathcal{P}_{n, k}\left (\mathcal{P}_{n,k}\left (D, \left ( C, C^{\prime}\right ), j\right )_\bot \right )_\bot &\,{\rightarrow }\, \mathcal{P}_{n, k}\left (D, \left ( C, C^{\prime}\right ), j\right )_\bot \end{eqnarray}

(46) \begin{eqnarray} \overline{\eta }_{\left ( D, \left ( C, C^{\prime}\right ), j \right )} \;:\; &\left ( D, \left ( C, C^{\prime}\right ), j \right ) &\,{\rightarrow }\,\mathcal{P}_{n, k}\left (D, \left ( C, C^{\prime}\right ), j\right )_\bot . \end{eqnarray}

in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ . Therefore, $\overline{\mathrm{m}}$ and $\overline{\eta }$ define the multiplication and the unit for $\mathcal{P}_{n, k}\left (-\right )_\bot$ , completing the definition of our monad. Analogously, we lift, as morphisms of $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k}\right )$ , the strength of $\left ( -\right )_\bot$ , making $\mathcal{P}_{n, k}\left (-\right )_\bot$ into a strong monad (i.e., $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k}\right )$ -enriched monad).

In order to finish the proof that $\left ( \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right ), \mathcal{P}_{n, k}\left (-\right )_\bot \right )$ is a $CBV$ $\boldsymbol{\omega } \mathbf{Cpo}$ -pair, it is enough to see that, for any pair of objects $\left ( D_0, \left ( C_0, C'_{\!\!0}\right ), j_0 \right )$ , $ \left ( D_1, \left ( C_1, C^{\prime}_1\right ), j_1 \right )$ of $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ , the least morphism $\bot \;:\; \left ( C_0, C'_{\!\!0}\right )\,{\rightarrow }\, \left ( \left ( C_1\right )_\bot, \left ( C^{\prime}_1\right )_\bot \right ), $ of $\boldsymbol{\omega } \mathbf{Cpo}\left ( C_0, \left ( C_1\right )_\bot \right ) \times \boldsymbol{\omega } \mathbf{Cpo}\left ( C'_{\!\!0}, \left (C^{\prime}_1\right )_\bot \right )$ defines the least morphism $\left ( D_0, \left ( C_0, C'_{\!\!0}\right ), j_0 \right )\,{\rightarrow }\, \mathcal{P}_{n, k}\left (D_1, \left ( C_1, C^{\prime}_1\right ), j_1\right )_\bot$ in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ .

Finally, since the underlying endofunctor of the monad $\mathcal{P}_{n, k}\left (-\right )_\bot$ , the multiplication and the identity are clearly lifted from $\left ( -\right )_\bot$ through $\underline{\mathcal{L}} _{n,k}$ as defined above, we have:

Therefore, by Lemma 5.2, $\mathcal{U}_{\mathcal{BV}} \left ( \underline{\mathcal{L}} _{n, k} \right )$ is a $CBV$ model morphism between the underlying $CBV$ models of $\left (\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right ), \mathcal{P}_{n, k}\left (-\right )_\bot \right )$ and $\left (\boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}, \left ( -\right )_\bot \right )$ .

9.3 Logical relations for $\mathbf{real}$ and deriving a $CBV$ model morphism

Henceforth, we assume that the macro $\mathcal{D}$ is sound for primitives (see Definition 7.5). We establish the $CBV$ model morphism (55). We start by establishing the logical relations’ assignment.

Let $(n, k)\in \mathbb{N}\times \left ( \mathbb{N} \cup \left \{ \infty \right \}\right )$ . We define the object (47) in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )$ .

(47) \begin{equation} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n,k} \stackrel{\mathrm{def}}{=} \left (\left \{ \left ( f \;:\; \mathbb{R} ^n\,{\rightarrow }\, \mathbb{R}, f^{\ast } \right ) \;:\; f\mbox{ is differentiable, } f^{\ast } = \mathfrak{D}^{k}{f} \right \}, \left ( \mathbb{R}, \mathbb{R} \times \mathbb{R} ^k \right ), \mathrm{incl.} \right ) \end{equation}

For each $m\in \mathbb{N}$ , $\mathrm{op} \in \mathrm{Op} _m$ , and $c\in \mathtt{R}$ , we define the morphisms (48), (49), and (50) in $\boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}$ , in which $\mathbb{D}$ , $[\![-]\!]$ , and $[\![-]\!] _{k}$ are the functors underlying the $CBV$ model morphisms, respectively, defined in (20), (30), and (32).

(48) \begin{align} \overline{[\![\hspace{-2.5pt}[\mathbf{sign}\, ]\!] \hspace{-2.5pt}]}_{k}\,&\stackrel{\mathrm{def}}{=} & \left ( \mathsf{sign}, \mathfrak{d}^{k}\left ( \mathsf{sign} \right ) \right ) = \left ( \mathsf{sign}, [\![\mathbb{D}\left ( \mathbf{sign}\, \right ) ]\!] _{k} \right ) \;:\; \left ( \mathbb{R}, \mathbb{R}\times \mathbb{R} ^k \right )\,{\rightarrow }\, \left ( \left (\mathsf{1}\sqcup \mathsf{1} \right )_\bot, \left ( \mathsf{1}\sqcup \mathsf{1}\right )_\bot \right ) \end{align}

(49) \begin{eqnarray} \overline{[\![\hspace{-2.5pt}[\underline{c} ]\!] \hspace{-2.5pt}]}_{k}&\stackrel{\mathrm{def}}{=} & \left ( \mathsf{c}, \mathfrak{d}^{k}\left ( \mathsf{c} \right ) \right ) \;:\; \left ( \mathsf{1}, \mathsf{1} \right )\,{\rightarrow }\, \left ( \mathbb{R}, \mathbb{R}\times \mathbb{R} ^k \right )\qquad\qquad\qquad\qquad\qquad \end{eqnarray}

(50) \begin{eqnarray} \overline{[\![\hspace{-2.5pt}[\mathrm{op} ]\!] \hspace{-2.5pt}]}_{k}&\stackrel{\mathrm{def}}{=} & \left ( [\![\mathrm{op} ]\!], \mathfrak{d}^{k}\left ( [\![\mathrm{op} ]\!] \right ) \right ) \;:\; \left ( \mathbb{R} ^m, \left (\mathbb{R}\times \mathbb{R}^k\right )^m \right )\,{\rightarrow }\, \left ( \left ( \mathbb{R}\right )_\bot, \left ( \mathbb{R}\times \mathbb{R} ^k\right ) _\bot \right ) \end{eqnarray}

By Proposition 8.4, we have that the product $ \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n,k} ^m$ in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right )$ is given by (51). Therefore, by the chain rule for derivatives, we have that (48), (49), and (50), respectively, define the morphisms (52), (53), and (54) in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )$ , where $\overline{\mathsf{1}}\sqcup \overline{\mathsf{1}}$ denotes the coproduct of the terminal $\overline{\mathsf{1}} = \left ( \mathsf{1}, \left ( \mathsf{1}, \mathsf{1} \right ), \mathrm{id} \right )$ with itself.

(51) \begin{eqnarray} && \left (\left \{ \left ( f_j \;:\; \mathbb{R} ^n\,{\rightarrow }\, \mathbb{R}, f^{\ast }_j\right )_{j\in \mathbb{I}_{m} } \;:\; f^{\ast }_j\mbox{ is differentiable and }\; f^{\ast }_j = \mathfrak{D}^{k}{f_j}, \forall j\in \mathbb{I}_{m} \right \}, \left ( \mathbb{R}, \mathbb{R}\times \mathbb{R}^k \right ) ^m, \mathrm{incl.} \right ) \nonumber \\ && \,{\cong}\, \left (\left \{ \left ( f \;:\; \mathbb{R} ^n\,{\rightarrow }\, \mathbb{R} ^m, f^{\ast }\right ) :\; f\mbox{ is differentiable, } f^{\ast } = \mathfrak{D}^{k}{f} \right \}, \left ( \mathbb{R} ^m, \left (\mathbb{R}\times \mathbb{R} ^k\right ) ^m \right ), \mathrm{incl.} \right ) . \end{eqnarray}

(52) \begin{equation} \overline{[\![\hspace{-2.5pt}[\mathbf{sign}\, ]\!] \hspace{-2.5pt}]}_{n, k} \;:\; \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n, k}\,{\rightarrow }\, \mathcal{P}_{n, k}\left (\overline{\mathsf{1}}\sqcup \overline{\mathsf{1}}\right )_\bot \end{equation}

(53) \begin{equation} \overline{[\![\hspace{-2.5pt}[\underline{c} ]\!] \hspace{-2.5pt}]}_{n,k} \;:\; \overline{\mathsf{1}}\,{\rightarrow }\, \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n,k} \end{equation}

(54) \begin{equation} \overline{[\![\hspace{-2.5pt}[\mathrm{op} ]\!] \hspace{-2.5pt}]}_{n,k} \;:\; \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n, k} ^m\,{\rightarrow }\, \mathcal{P}_{n,k}\left ( \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n,k} \right )_\bot \end{equation}

By the universal property of the $CBV$ model $\left ( \mathbf{Syn}_V, \mathbf{Syn}_{\mathcal{S}},\mathbf{Syn}_\mu, \mathbf{Syn}_{\mathsf{it}} \right )$ , we get:

Proposition 9.4. For each $(n, k)\in \mathbb{N}\times \left ( \mathbb{N} \cup \left \{ \infty \right \}\right )$ , there is only one $CBV$ model morphism

(55) \begin{equation} \overline{[\![\hspace{-2.5pt}[-]\!] \hspace{-2.5pt}]}_{n,k} :\left ( \mathbf{Syn}_V^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathcal{S}}^{{\mathbf{tr}}},\mathbf{Syn}_\mu ^{{\mathbf{tr}}}, \mathbf{Syn}_{\mathsf{it}}^{{\mathbf{tr}}} \right )\,{\rightarrow }\, \mathcal{U}_{\mathcal{BV}} \left ( \mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n, k} \right ), \mathcal{P}_{n, k}\left (-\right )_\bot \right ) \end{equation}

that is consistent with the assignment given by (47), (52), (54), and (53). Moreover, Diag. (56) commutes.

(56)

9.4 AD logical relations for data types

As a consequence of Proposition 9.4, we establish a fundamental result on the logical relations $ \overline{[\![\hspace{-2.5pt}[-]\!] \hspace{-2.5pt}]}_{n,k}$ for data types (i.e., types not containing function types) in our setting: namely, Proposition 9.6. Observe that, by distributivity of products over coproducts, any such data type is isomorphic to $\bigsqcup _{j\in L}\mathbf{real}^{l_j}$ for some finite set $L$ and $l_j\in \mathbb{N}$ . Therefore, we start by establishing Lemma 9.5 about our logical relations and the coproducts in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )$ .

Proof By Proposition 9.2, $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{n,k} \right )$ has coproducts. Moreover, we can conclude that $\displaystyle \left ( g, \dot{g}\right ) \in \coprod _{j\in L} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{n,k} ^{l_j}$ implies that, for some $r\in L$ , we have a pair

(57) \begin{equation} \displaystyle \left (\underline{g} \;:\; \mathbb{R}^{n}\,{\rightarrow }\,\mathbb{R} ^{l_r}, \mathfrak{D}^{k}{g} \;:\; \left (\mathbb{R}\times \mathbb{R} ^k\right ) ^{n}\,{\rightarrow }\, \left (\mathbb{R}\times \mathbb{R} ^k\right ) ^{l_r} \right ) \end{equation}

such that $\left ( g, \dot{g}\right ) = \left ( \iota _{\mathbb{R} ^{l_r} }\circ \underline{g}, \iota _{\left (\mathbb{R}\times \mathbb{R} ^k\right ) ^{l_r} }\circ \mathfrak{D}^{k}{g} \right )$ . Following Definition 7.2, this completes our proof.

Corollary 9.7. Let $k\in \mathbb{N}\cup \left \{\infty \right \}$ . If, for each $i\in \mathfrak{L}$ , the morphism $ \left (g, \dot{g} \right )$ in $\boldsymbol{\omega } \mathbf{Cpo}\times \boldsymbol{\omega } \mathbf{Cpo}$ defines the morphism (60) in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{{s_i}, k} \right )$ , then $\displaystyle g \;:\; \coprod _{r\in \mathfrak{L} } \mathbb{R} ^{s_r}\,{\rightarrow }\,\left ( \coprod _{j\in L} \mathbb{R} ^{l_j}\right )_\bot$ is differentiable and $\dot{g} = \mathfrak{d}^{k}\left ( g\right )$ .

(60) \begin{equation} \mathtt{g} \;:\; \coprod _{r\in \mathfrak{L} } \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{s _r}\,{\rightarrow }\, \mathcal{P}_{{s_i}, k}\left (\coprod _{j\in L} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{l _j}\right )_\bot \end{equation}

(61) \begin{equation} \iota _{i} \;:\; \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{s _i}\,{\rightarrow }\, \coprod _{r\in K} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{s _r} \end{equation}

Proof From the hypothesis, for each $i\in \mathfrak{L}$ , we conclude that the pair (62) defines the morphism (64), since $\left ( \iota _{\mathbb{R}^{s_i}}, \iota _{\left ( \mathbb{R}\times \mathbb{R} ^k\right ) ^{s_i}} \right )$ defines the coprojection (61) in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{{s_i}, k} \right )$ .

(62) \begin{equation} \left ( g_i\stackrel{\mathrm{def}}{=} g\circ \iota _{\mathbb{R}^{s_i}}, \, \dot{g}_i\stackrel{\mathrm{def}}{=} \dot{g} \circ \iota _{\left ( \mathbb{R}\times \mathbb{R} ^k\right ) ^{s_i}} \right ) \end{equation}

(63) \begin{equation} \mathtt{g}_i \stackrel{\mathrm{def}}{=} \mathtt{g}\circ \iota _{i} \;:\; \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{s _i}\,{\rightarrow }\, \mathcal{P}_{{s_i}, k}\left (\coprod _{j\in L} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{l _j}\right )_\bot \end{equation}

Since $\mathrm{id} _{\mathbb{R} ^{s_i}} \;:\; \mathbb{R} ^{s_i}\,{\rightarrow }\, \mathbb{R}^{s_i}$ is differentiable, and $\mathfrak{D}^{k}{\left ( \mathrm{id} _{\mathbb{R} ^{s_i}} \right ) }$ is given by the identity $\left ( \mathbb{R}\times \mathbb{R} ^k\right ) ^{s_i}\,{\rightarrow }\,\left ( \mathbb{R}\times \mathbb{R} ^k\right ) ^{s_i}$ , we conclude that

(64) \begin{equation} \left (g _i,\dot{g}_i \right )\in \underline{\mathcal{P}_{{s_i},k}\left (\coprod _{j\in L} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{l_j}\right )_\bot }. \end{equation}

By Proposition 9.6, (64) proves that $g_i$ is differentiable and $\dot{g}_i = \mathfrak{d}^{k}\left ( g_i\right )$ . Since this result holds for any $i\in \mathfrak{L}$ , we conclude that $g$ is differentiable and $\dot{g} = \mathfrak{d}^{k}\left ( g \right )$ .

9.5 Fundamental AD correctness theorem

We prove Theorem 9.8, which completes the proof of Theorem 9.1.

Proof We assume that we have $t$ as above. For each $i\in \mathfrak{L}$ , the pair (65) is in the image of $\left ([\![-]\!] \times [\![-]\!] _{k} \right )\circ \left ( \mathrm{id}\times \mathbb{D}\right ) =\mathcal{U}_{\mathcal{BV}} \left ({\underline{\mathcal{L}}}_{{s_i},k}\right )\circ \overline{[\![\hspace{-2.5pt}[-]\!] \hspace{-2.5pt}]}_{{s_i},k}$ . This implies that (65) defines the morphism (66) in $\mathbf{Sub}\left ( \boldsymbol{\omega } \mathbf{Cpo}\downarrow G_{{s_i}, k} \right )$ . Therefore, by Corollary 9.7, we conclude that $[\![t]\!]$ is differentiable and $[\![\mathbb{D}{\left ( t\right ) } ]\!] _{k} = \mathfrak{d}^{k}\left ( [\![t]\!] \right )$ .

(65) \begin{equation} \left ([\![t]\!], [\![\mathbb{D}{\left ( t\right ) } ]\!] _{k} \right ) \end{equation}

(66) \begin{equation} \overline{[\![\hspace{-2.5pt}[t]\!] \hspace{-2.5pt}]}_{{s_i},k} \;:\; \coprod _{r\in K} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{s _r}\,{\rightarrow }\, \mathcal{P}_{{s_i}, k}\left (\coprod _{j\in L} \overline{[\![\hspace{-2.5pt}[\mathbf{real} ]\!] \hspace{-2.5pt}]}_{{s_i},k} ^{l _j}\right )_\bot \end{equation}

9.6 Correctness of the dual numbers forward AD

We assume that $\mathbf{vect}$ implements the vector space $\mathbb{R}$ . It is straightforward to see that we get forward mode AD out of our macro $\mathcal{D}$ : namely, for a program $x:\tau \vdash{t}:\sigma$ (where $\tau$ and $\sigma$ are data types) in the source language, we get a program $x:\mathcal{D}\;{(\tau )} \vdash \mathcal{D}\;{(t)}:\mathcal{D}\;{(\sigma )}$ in the target language, which, by Theorem 9.1, satisfies the following properties:

• • $[\![{t} ]\!] \;:\; \coprod _{r\in K } \mathbb{R} ^{n_r}\,{\rightarrow }\,\left ( \coprod _{j\in L } \mathbb{R} ^{m_j} \right )_\bot$ is differentiable as in Definition 7.5;

• • if $y\in \mathbb{R} ^{n_i}\cap [\![{t} ]\!] ^{-1}\left ( \mathbb{R} ^{m_j} \right ) = W_ j$ for some $i\in K$ and $j\in L$ , we have that, for any $w\in \mathbb{R} ^{n_i}$ , denoting $z := {\phi} _{{n_i},1}\left (y,w\right )$ ,

  (67) \begin{eqnarray} [\![ \mathcal{D}\;{(t)} ]\!] _{1} \left ( {\phi} _{{n_i},1}\left (y,w\right ) \right ) &=& \mathfrak{d}^{1}\left ( [\![t]\!] \right ) \left ( z\right ) = \mathfrak{D}^{1}{[\![t]\!] |_{W_j} } \left ( z\right ) = {\phi} _{{m_j},1}\left ( [\![{t} ]\!] \left ( y\right ), \tilde{w}\cdot [\![{t} ]\!] '(y) ^{t} \right ) \nonumber \\ &=& {\phi} _{l,1}\left ( [\![{t} ]\!] \left ( y\right ), [\![{t} ]\!] '(y)(w) \right ), \end{eqnarray}
  where $[\![{t} ]\!] '(y) \;:\; \mathbb{R} ^{n_i}\,{\rightarrow }\,\mathbb{R} ^{m_j}$ is the derivative of $[\![{t} ]\!] |_{W_j}\;:\; W_j\,{\rightarrow }\, \mathbb{R} ^{m_j}$ at $y$ .

9.7 Correctness of the dual numbers reverse AD

We assume that $\mathbf{vect}$ implements the vector space $\mathbb{R} ^k$ , for some fixed $k\in \mathbb{N}\cup \left \{\infty \right \}$ . We consider the respective (co)projections $\mathfrak{p}_{k \rightarrow s}$ for each $s\in \mathbb{N}\cup \left \{\infty \right \}$ , as defined in (18) . The following shows how our macro encompasses reverse-mode AD.

For each $s\in \mathbb{N} ^\ast$ with $s \leq k$ , we can define the morphism $\mathbf{wrap}_{s}\stackrel{\mathrm{def}}{=} \left ( \pi _{j}, \overline{e} _{j} \right ) _{j\in \mathbb{I}_{s}} \;:\; \mathbf{real} ^s\,{\rightarrow }\, \left ( \mathbf{real}\times \mathbf{vect} \right ) ^s$ in $\mathbf{Syn}_V^{{\mathbf{tr}}}$ , which corresponds to the wrapper defined in (2) in the target language. We denote $\mathtt{wrap}_{s}\stackrel{\mathrm{def}}{=}[\![\mathbf{wrap}_{s}]\!] _{k}$ . By the definition of the $k$ -semantics, it is clear that $\mathtt{wrap}_{s} \left ( y \right ) = {\phi} _{s,k}\left ( y, e^{k}_{1}, \ldots, e^{k}_{s} \right )$ .

For a program $x:\mathbf{real}^{s} \vdash{t}:\mathbf{real} ^l$ (where $s, l\in \mathbb{N}^\ast$ ), we have that, for any $y\in [\![{t} ]\!] ^{-1}\left ( \mathbb{R} ^l \right )\subset \mathbb{R} ^s$ ,

\begin{eqnarray*} [\![\mathcal{D}\;{(t)}\circ \mathbf{wrap}_{s} ]\!] _{k} \left ( y \right ) & = & \mathfrak{d}^{k}\left ( [\![t]\!] \right ) \circ \mathtt{wrap}_{s}\left ( y \right ) = \mathfrak{D}^{k}{[\![t]\!] } \circ \mathtt{wrap}_{s}\left ( y \right )\\ & = & \mathfrak{D}^{k}{[\![t]\!] } \circ {\phi} _{s,k}\left ( y, e^{k}_{1}, \ldots, e^{k}_{s} \right ) \\ &=& {\phi} _{l,k}\left ( [\![{t} ]\!] \left ( y\right ), \mathfrak{p}_{s \rightarrow k} [\![{t} ]\!] '(y) ^t \right ) \end{eqnarray*}

by Theorem 9.1. This gives the transpose derivative $\mathfrak{p}_{s \rightarrow k} [\![{t} ]\!] '(y) ^t$ as something of the type $\mathbf{vect} ^l$ . This should be good enough whenever $k = s$ , since, in this case, $[\![\mathbf{vect} ^l]\!] _{k} = \left ( \mathbb{R}^s\right ) ^l$ and $\mathfrak{p}_{s \rightarrow k} =\mathfrak{p}_{k \rightarrow k} = \mathrm{id}$ .

In case of $s \lt k$ , if needed, the type can be fixed by using the handler $\mathfrak{h} _{s}$ . More precisely, we can define the morphism