Remark 3.3. Although the Universal Approximation Theorem does not directly apply to non-constant classification functions in the class (2.1), if we consider a classification function restricted to a finite set (e.g., training and validation sets), it will have a continuous extension and hence the Universal Approximation Theorem will apply. Furthermore, recent results discuss existence theorems in the general setting of (2.1) [Reference Liu and Hansen53].

Remark 3.6 (Theorem 3.5 is independent of the exact computational model). Note that the result above is independent of whether the underlying computational device is a BSS machine or a Turing machine. To achieve this, we work with a definition of an algorithm termed a general algorithm. The corresponding definitions as well as a formal statement of Theorem 3.5 are detailed inSection 5.4 and Proposition 5.26, respectively.

Remark 3.7 (Irrelevance of local minima). Note that Theorem 3.5 has nothing to do with the potential issue of the optimisation problem having several local minima. Indeed, the general algorithms used in Theorem 3.5 are more powerful than any Turing machine or BSS machine as will be discussed further in Remark 5.13.

Remark 3.8 (Hilbert’s 10th problem). Finally, we mention in passing that Theorem 3.5 demonstrates – in contrast to Hilbert’s 10th problem [57] – that non-computability results in DL do not prevent provable existence results. Indeed, because of the implication in (3.1) and the non-computability of Hilbert’s 10th problem [Reference Matiyasevich57, Reference Poonen68] (when $\Theta$ is the collection of all polynomials with integer coefficients in Example 3.2), there are infinitely many Diophantine equations for which one cannot prove existence of an integer solution – or a negation of the statement.

Remark 5.1. Although we focus on the ReLU non-linearity, it is possible to use the techniques presented in this paper to prove similar results for other non-linearities like the leaky ReLU [55] $\rho ^{\text{leaky}}$ and the parameterised ReLU [46] $\rho ^{param}_{\alpha }$ where

\begin{align*} \rho ^{\text{leaky}}(x) = \begin{cases} 0.01 \cdot x & \text{if } x \lt 0\\ x & \text{ if } x \geq 0 \end{cases}, \quad \rho ^{param}_{\alpha }(x) = \begin{cases} \alpha x & \text{if } x \lt 0\\ x & \text{ if } x \geq 0 \end{cases} \end{align*}

Proof. Throughout this proof, we will use the convention that for a random variable $Y\,:\,\Omega \to \mathcal{E}$ the notation $\{Y = \mu \}$ for $\mu \in \Omega$ means the set $\{\tau \in \Omega \, \vert \, Y(\tau ) = \mu \}$ .

For item (i), define the random variable $M_\theta$ to be the number of different unique values taken by the random variables $\lceil X_1/2 \rceil$ ,…, $ \lceil X_\theta /2 \rceil$ and note that $ {\mathbb{P}}\left ( N\lt \beta \right )\leq {\mathbb{P}} (M_r \lt \beta )$ , for $\beta \in \mathbb{R}$ . Now, as the random variables $\lceil X_j/2 \rceil$ , $j=1,\ldots ,r$ , are i.i.d. and distributed according to the zeta distribution with parameter $\gamma$ , it follows from [Reference Zakrevskaya and Kovalevskii82, Lemmas 4, 3] that $ \mathbb{E}[M_\theta ]\gt (1-e^{-C_\zeta (\gamma )}) \theta ^{1/\gamma }$ and $ \sigma ^2\,:\!=\,\mathrm{Var}[M_\theta ]\leq \mathbb{E}[M_\theta ] \leq \theta$ , and hence Chebyshev’s inequality yields

\begin{align*} &{\mathbb{P}}\left ( N\lt \frac {1-e^{-C_\zeta (\gamma )}}{2} \theta ^{1/\gamma } \right )\leq {\mathbb{P}} \left (M_r \lt \frac {1-e^{-C_\zeta (\gamma )}}{2} \theta ^{1/\gamma } \right ) \\ \leq \;& {\mathbb{P}} \left (|M_\theta -\mathbb{E}[M_\theta ]| \gt \frac {1-e^{-C_\zeta (\gamma )}}{2\sigma } \theta ^{1/\gamma } \cdot \sigma \right )\leq \left (\frac {1-e^{-C_\zeta (\gamma )}}{2\sigma } \theta ^{1/\gamma } \right )^{-2}\leq \frac {4 \theta ^{-(2/\gamma -1)}}{(1-e^{-C_\zeta (\gamma )})^{2}} , \end{align*}

which implies item (i).

The proof of item (ii) is simple. Note that $\{\max S\leq n\}=\bigcap _{j=1}^r\{X_j\leq n\}$ and, for each $j$ ,

\begin{align*} {\mathbb{P}}(X_j \leq n)=\sum _{j=1}^n \mathrm{p}_j &\geq \sum _{j=1}^{\lfloor n/2 \rfloor } C_\zeta (\gamma )j^{-\gamma }\geq 1- C_\zeta (\gamma )\int _{\lfloor n/2 \rfloor }^\infty t^{-\gamma }\mathrm{d}t \geq 1 - \frac {C_\zeta (\gamma )}{\gamma -1} \lfloor n/2 \rfloor ^{1-\gamma }, \end{align*}

and hence, as the $X_j$ are independent,

\begin{align*} {\mathbb{P}}(\! \max S\leq n)= {\mathbb{P}}(X_j \leq n)^\theta \geq \left (1 - \frac {C_\zeta (\gamma )}{\gamma -1} \lfloor n/2 \rfloor ^{1-\gamma }\right )^\theta \geq 1- \frac {C_\zeta (\gamma )}{\gamma -1} \theta \lfloor n/2 \rfloor ^{1-\gamma } \end{align*}

where the last inequality follows by Bernoulli’s inequality.

Item (iii) is somewhat more involved. We start by outlining the strategy: the set $S$ may contain pairs of the form $(Z_{j},Z_{j+1})=(2i-1,2i)$ , that is, an odd natural number followed by the next even one. We will condition on the set of $j$ where $(Z_j,Z_{j+1})$ is such a pair, as well as the specific value of $Z_j$ .

More precisely, for fixed sets $\mathcal{I}$ and $\mathcal{J}$ with $|\mathcal{I}| = |\mathcal{J}|$ , enumerated by

\begin{align*} \mathcal{I} = \{i_1,i_2,\dotsc ,i_m\} \text{ and } \mathcal{J} = \{j_1,j_2,\dotsc ,j_m\}, \end{align*}

let $\mathcal{A} = \{1,\ldots , N\} \setminus \big ( \mathcal{J} \cup (\mathcal{J}+1)\big )$ where $\mathcal{J}+1\,:\!=\,\{j+1\,\vert \,j\in \mathcal{J}\}$ . We will condition on the event $F_{\mathcal{I},\mathcal{J}}$ which occurs precisely when $N = n$ , $(Z_{j_\ell },Z_{j_\ell +1})=(2i_\ell -1,2i_\ell )$ for $\ell \in \{1,2,\dotsc ,m\}$ , and, on the indices in $\mathcal{A}$ , the set $S$ contains no odd–even pairs, that is, $(Z_{a},Z_{a+1})\notin \{ (2i-1,2i)\,\vert \, i\in \mathbb{N}\}$ for all $a \in \mathcal{A}$ with $a \lt n$ and $(Z_{a-1},Z_{a})\notin \{ (2i-1,2i)\,\vert \, i\in \mathbb{N}\}$ for all $a \in \mathcal{A}$ with $a \gt 1$ . With varying $\mathcal{I}$ and $\mathcal{J}$ , these sets $F_{\mathcal{I},\mathcal{J}}$ partition the event $\{N=n\}$ .

The intuition behind this construction is as follows: conditional on $F_{\mathcal{I},\mathcal{J}}$ , whenever $j \in \mathcal{J}$ we have $Z_{j+1} - Z_j = 1$ and hence $\chi _{\{Z_{j+1}-Z_{j}\text{ odd}\}} = 1$ . Thus for sets $\mathcal{J}$ with $|\mathcal{J}| \geq n/5$ , we are done. If instead $|\mathcal{J}|$ is small, then $|\mathcal{A}|$ will be relatively large. For $a \in \mathcal{A}$ , we will argue that every $Z_a$ has equal probability of being an odd number or the even number following it, owing to the assumption that $\mathrm{p}_{2i-1}=\mathrm{p}_{2i}$ and the assumption that if $a \lt n$ then $(Z_{a},Z_{a+1})\notin \{ (2i-1,2i)\,\vert \, i\in \mathbb{N}\}$ and if $a \gt 1$ then $(Z_{a-1},Z_{a})\notin \{ (2i-1,2i)\,\vert \, i\in \mathbb{N}\}$ .

This will allow us to conclude that the indicator random variables $\chi _{\{Z_{a}\text{ odd}\}}$ for $a\in \mathcal{A}$ are independent symmetric Bernoulli random variables (that is to say, they take the values $1$ and $0$ each with probability $1/2$ ). The desired bound will follow by an application of Hoeffding’s inequality.

We are now ready to present the formal proof. If $\theta \lt 10$ there is nothing to prove, so assume that $\theta \geq 10$ and fix an $n$ such that $10\leq n\leq \theta$ . Consider arbitrary sets $\mathcal{I}\subset \mathbb{N}$ and $\mathcal{J}\subset \{1,\ldots ,n-1\}$ so that

(5.5) \begin{equation} m\,:\!=\,|\mathcal{I}|=|\mathcal{J}|\lt n \text{ and } \mathcal{J}\cap (\mathcal{J}+1)=\varnothing , \end{equation}

and define $\mathcal{A}\,:\!=\,\{1,\ldots , N\} \setminus \big ( \mathcal{J} \cup (\mathcal{J}+1) \big )$ . Enumerate $\mathcal{I}=\{i_{1},\dotsc ,i_{m}\}$ with $i_{1}\lt \dotsb \lt i_{m}$ , $\mathcal{J}=\{j_1,\dotsc ,j_m\}$ with $j_1\lt \dotsb \lt j_m$ , and $\mathcal{A}=\{a_1,\dotsc ,a_{n-2m}\}$ with $a_1\lt \dotsb \lt a_{n-2m}$ and define the event

\begin{align*} \begin{split} F_{\mathcal{I},\mathcal{J}}=\{N=n\}\cap \bigcap _{\ell =1}^m\{(Z_{j_\ell }, Z_{j_\ell +1})&=(2i_\ell -1,2i_\ell )\} \cap \bigcap _{\substack {a\in \mathcal{A}, a\lt n\\ i\in \mathbb{N}}}\{(Z_{a},Z_{a+1})\neq (2i-1,2i)\}\\ &\qquad \cap \bigcap _{\substack {a\in \mathcal{A}, 1\lt a \leq n\\ i\in \mathbb{N}}}\{(Z_{a-1},Z_{a})\neq (2i-1,2i)\}. \end{split} \end{align*}

Note that, for every $n\in \mathbb{N}$ , we have

(5.6) \begin{equation} \{N=n\}={\bigcup _{ \substack {\mathcal{I}\subset \mathbb{N}, \mathcal{J}\subset \{1,\ldots ,n-1\}\\\text{satisfying (5.5)}}}} F_{\mathcal{I},\mathcal{J}}, \end{equation}

that is, the events $F_{\mathcal{I},\mathcal{J}}$ for different $\mathcal{I}$ and $\mathcal{J}$ partition the event $\{N=n\}$ , and thus our strategy will be to prove the bound $ {\mathbb{P}}\left (\left .\sum _{j=1}^{N-1} \chi _{\{Z_{j+1}-Z_{j}\text{ odd}\}} \leq n/5 \,\right \rvert \, F_{\mathcal{I},\mathcal{J}} \right ) \leq e^{-n/100}$ for each of these events.

The argument relies on bounding from below the number of indices $j$ such that $Z_{j+1}-Z_{j}$ is odd. For $j\in \mathcal{J}$ , this will be easy, as $Z_{j+1}-Z_j=2i_j-(2i_j-1)=1$ is always odd, by definition of $F_{\mathcal{I},\mathcal{J}}$ . For $j\in \mathcal{A}$ , we will need the following claim which we prove last.

Armed with the claim, the counting argument is as follows. Note that, on the event $F_{\mathcal{I},\mathcal{J}}$ , for $k\in \{1,\ldots , n-2m-1\}$ such that $a_{k+1}\gt a_k+1$ , we have that $\{Z_{a_{k}},\ldots , Z_{a_{k+1}}\}=\{Z_{a_k}, 2i_{t}-1,2i_{t},2i_{t+1}-1,2i_{t+1},\dotsc , 2i_{t+s-1} -1,2i_{t+s-1},Z_{a_{k+1}}\}$ for some $t \in \{1,2,\dotsc ,m\}$ and where $s=|\mathcal{J}\cap \{a_k,\ldots , a_{k+1}-1\}|$ . Hence,

(5.7) \begin{align} \sum _{\ell =a_k}^{a_{k+1}-1} \chi _{\{Z_{\ell +1}-Z_{\ell }\text{ odd}\}}&\geq \chi _{\{2i_{t}-1-Z_{a_k}\text{ odd}\}} + \sum _{\ell =0}^{s-1} \chi _{\{ (2i_{t+\ell }) - (2i_{t+\ell }-1) \text{ odd}\}} + \chi _{\{Z_{a_{k+1}}-2i_{t+s-1}\text{ odd}\}}\notag \\ & = \chi _{\{Z_{a_k}\text{ even}\}} + |\mathcal{J}\cap \{a_k,\ldots , a_{k+1}-1\}| + \chi _{\{Z_{a_{k+1}}\text{ odd}\}}\notag \\ & \geq |\mathcal{J}\cap \{a_k,\ldots , a_{k+1}-1\}| + \chi _{\{Z_{a_{k+1}}-Z_{a_k}\text{ even}\}}, \end{align}

where we used the simple observation that $\chi _{\{Z_{a_k}\text{ even}\}} + \chi _{\{Z_{a_{k+1}}\text{ odd}\}}\geq \chi _{\{Z_{a_{k+1}}-Z_{a_k}\text{ even}\}}$ . This motivates defining random variables $E_{a_k}$ with $k\in \{1,\ldots , n-2m-1\}$ conditioned on the event $F_{\mathcal{I},\mathcal{J}}$ according to

\begin{align*} E_{a_k}& =\begin{cases} 1,& Z_{a_{k}+1}-Z_{a_k}\text{ is odd}\\[5pt] 0, & Z_{a_{k}+1}-Z_{a_k}\text{ is even} \end{cases}, \quad \text{for $k$ s.t. $a_{k+1}=a_k+1$, and}\\ E_{a_k}&=\begin{cases} 0,& Z_{a_{k+1}}-Z_{a_k}\text{ is odd}\\ 1, & Z_{a_{k+1}}-Z_{a_k}\text{ is even} \end{cases}, \quad \text{for $k$ s.t. $a_{k+1}\gt a_k+1$,} \end{align*}

which, as a consequence of the Claim, are themselves independent symmetric Bernoulli random variables. Thus, writing $U\,:\!=\,\sum _{k=1}^{N-1} \chi _{\{Z_{k+1}-Z_{k}\text{ odd}\}}$ , on the event $F_{\mathcal{I},\mathcal{J}}$ we have

(5.8) \begin{align} U&=\sum _{\ell \lt a_1\text{ or }\, \ell \geq a_{n-2m}} \chi _{\{Z_{\ell +1}-Z_{\ell }\text{ odd}\}} + \sum _{k=1 }^{n-2m-1}\sum _{\ell =a_k}^{a_{k+1}-1} \chi _{\{Z_{\ell +1}-Z_{\ell }\text{ odd}\}} \notag \\ &\geq |\mathcal{J}\cap \{1,\ldots , a_1-1\}| + |\mathcal{J}\cap \{a_{n-2m}, \ldots , n\} | \notag \\ &\qquad \qquad + \sum _{\substack {1\leq k\leq n-2m-1\\a_{k+1}=a_k+1}} \chi _{\{Z_{a_{k}+1}-Z_{a_k}\text{ odd}\}} + \sum _{\substack {1\leq k\leq n-2m-1\\a_{k+1}\gt a_k+1}}\sum _{\ell =a_k}^{a_{k+1}-1} \chi _{\{Z_{\ell +1}-Z_{\ell }\text{ odd}\}} \notag \\ &\geq |\mathcal{J}\cap \{1,\ldots , a_1-1\}| + |\mathcal{J}\cap \{a_{n-2m}, \ldots , n\} | \notag \\ &\qquad \qquad + \sum _{\substack {1\leq k\leq n-2m-1\\a_{k+1}=a_k+1}} E_{a_k} + \sum _{\substack {1\leq k\leq n-2m-1\\a_{k+1}\gt a_k+1}}\left ( |\mathcal{J}\cap \{a_k ,\ldots , a_{k+1}-1\}| + E_{a_k} \right ) \notag \\ &=|\mathcal{J}| + \sum _{k=1}^{n-2m-1} E_{a_k}= m+ \sum _{k=1}^{n-2m-1} E_{a_k}, \end{align}

where the second inequality is due to (5.7) and the penultimate equality follows from the observation that $|\mathcal{J}\cap \{a_k ,\ldots , a_{k+1}-1\}| = 0$ whenever $a_{k+1} = a_k + 1$ .

Now, for sets $\mathcal{I}\subset \mathbb{N}$ and $\mathcal{J}\subset \{1,\ldots ,n-1\}$ satisfying (5.5) as well as $m=|\mathcal{I}|=|\mathcal{J}|\leq n/5$ , we have that (5.8) implies $U\geq \sum _{k=1}^{n-2m-1} E_{a_k}$ , which together with Hoeffding’s inequality yields

\begin{align*} {\mathbb{P}}\Big (U \leq n/5 \,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \Big )& \leq {\mathbb{P}}\Big ( \sum _{k=1}^{n-2m-1} E_{a_k} \leq n/5 \,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \Big )\\ &\leq \exp \left (-2\Big (\frac {1}{2} - \frac {n/5}{n-2m-1}\Big )^{2}(n-2m-1) \right )\leq \exp \left (- n/100 \right ) \end{align*}

where in the last inequality we used $n-2m-1\geq n/2$ (recall that $n\geq 10$ ). On the other hand, in the case when $m=|\mathcal{I}|=|\mathcal{J}|\gt n/5$ we have ${\mathbb{P}}\Big (U \leq n/5 \,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \Big )=0$ directly from (5.8).

Therefore, we have shown that for any $\mathcal{I}$ , $\mathcal{J}$ satisfying (5.5), ${\mathbb{P}}\Big (U \leq n/5 \,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \Big ) \leq \exp (\! -n/100)$ and so using (5.6)

\begin{align*} {\mathbb{P}}\Big ( U \leq n/5, N=n \Big )&= \sum _{ \substack {\mathcal{I}\subset \mathbb{N}, \mathcal{J}\subset \{1,\ldots ,n-1\}\\\text{satisfying (5.5)}}} {\mathbb{P}}\Big ( U\leq n/5 \,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \Big ) {\mathbb{P}}(F_{\mathcal{I},\mathcal{J}})\\ &\leq \exp \left (- n/100 \right ) {\mathbb{P}}\Bigg (\bigcup _{ \substack {\mathcal{I}\subset \mathbb{N}, \mathcal{J}\subset \{1,\ldots ,n-1\}\\\text{satisfying (5.5)}}} F_{\mathcal{I},\mathcal{J}} \Bigg ) = \exp \left (- n/100 \right ) {\mathbb{P}}(N=n), \end{align*}

which yields the desired bound after dividing both sides by ${\mathbb{P}}(N=n)$ .

It remains to prove the Claim. To this end, fix $n$ , $\mathcal{I}=\{i_1\lt \ldots \lt i_m\}$ , $\mathcal{J}=\{j_1\lt \ldots \lt j_m\}$ and $\mathcal{A}=\{a_1\lt \ldots \lt a_{n-2m}\}$ satisfying (5.5). Then, conditional on $F_{\mathcal{I},\mathcal{J}}$ we can write $Z_a=2\lceil Z_a/2 \rceil - \chi _{\{Z_{a}\text{ odd}\}}$ , for $a\in \mathcal{A}$ , where the $\chi _{\{Z_{a}\text{ odd}\}}$ are random variables taking values in $\{0,1\}$ and the $\lceil Z_a/2 \rceil$ are random variables taking values in $\mathbb{N}\setminus \mathcal{I}$ and moreover $\lceil Z_{a_1}/2 \rceil \lt \ldots \lt \lceil Z_{a_{n-2m}}/2 \rceil$ . Now, for a set $\mathcal{U}=\{u_1\lt \ldots \lt u_{n-2m}\}\subset \mathbb{N}\setminus \mathcal{I}$ denote $\mathring F_{\mathcal{U}}=\bigcap _{j=1}^{n-2m}\{\lceil Z_{a_j}/2 \rceil =u_j\}$ so that for any $b\in \{0,1\}^{n-2m}$

(5.9) \begin{align} &{\mathbb{P}}\left (\{\chi _{\{Z_{a_1}\text{ odd}\}}=b_1,\; \ldots \; , \chi _{\{Z_{a_{n-2m}}\text{ odd}\}}=b_{n-2m}\}\,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \right )\nonumber\\ &\quad=\,\sum _{\mathcal{U}\subset \mathbb{N}\setminus \mathcal{I}}{\mathbb{P}}\left (\{\chi _{\{Z_{a_1}\text{ odd}\}}=b_1,\; \ldots \; , \chi _{\{Z_{a_{n-2m}}\text{ odd}\}}=b_{n-2m}\}\,\Big \rvert \, F_{\mathcal{I},\mathcal{J}}\cap \mathring F_{\mathcal{U}} \right ) {\mathbb{P}}( \mathring F_{\mathcal{U}} \, \vert \, F_{\mathcal{I},\mathcal{J}}) \nonumber \\ &\quad=\,\sum _{\mathcal{U}\subset \mathbb{N}\setminus \mathcal{I}}{\mathbb{P}}\left (\{\chi _{\{Z_{a_1}\text{ odd}\}}=0,\; \ldots \; , \chi _{\{Z_{a_{n-2m}}\text{ odd}\}}=0\}\,\Big \rvert \, F_{\mathcal{I},\mathcal{J}}\cap \mathring F_{\mathcal{U}} \right ) {\mathbb{P}}( \mathring F_{\mathcal{U}} \, \vert \, F_{\mathcal{I},\mathcal{J}}) \\ &\quad=\,{\mathbb{P}}\left (\{\chi _{\{Z_{a_1}\text{ odd}\}}=0,\; \ldots \; , \chi _{\{Z_{a_{n-2m}}\text{ odd}\}}=0\}\,\Big \rvert \, F_{\mathcal{I},\mathcal{J}} \right ), \nonumber\end{align}

where in (5.9) we used the fact that $\mathrm{p}_{2j-1}=\mathrm{p}_{2j}$ , for all $j\in \mathbb{N}$ . It hence follows that the $\chi _{\{Z_{a_j}\text{ odd}\}}$ , $1\leq j\leq n-2m$ , conditional on $F_{\mathcal{I},\mathcal{J}}$ are independent symmetric Bernoulli variables, establishing the Claim and thus completing the proof.

Proof. We may w.l.o.g. assume that $K$ is divisible by 4. Indeed, if $K$ is not divisible by $4$ , we can extend the sequence $\{\alpha _k\}_{k=1}^{K}$ by adjoining two new elements (say $\alpha _{K}/2$ and $\alpha _K/4$ ) at the end of the sequence. We additionally set $\alpha _{K+1}=0$ for convenience. Now, for $\ell \in \{1,\ldots , K/4\}$ , define the single-layer neural network

\begin{align*} \begin{aligned} \psi _\ell (x)&=\left (\alpha _{4\ell -2}-\alpha _{4\ell -1}\right )^{-1} \big ( \rho (\alpha _{4\ell -2} -x_1)- \rho (\alpha _{4\ell -1} -x_1) \big )\\ &\qquad \qquad - \left (\alpha _{4\ell }-\alpha _{4\ell +1}\right )^{-1} \big ( \rho (\alpha _{4\ell } -x_1)- \rho (\alpha _{4\ell +1} -x_1) \big ),\quad \text{for }x\in {\mathbb{R}}^{N_0}. \end{aligned} \end{align*}

One now easily verifies that $\psi _{\ell }(x)=1$ whenever $x_1\in [\alpha _{4\ell }, \alpha _{4\ell -1}]$ and $\psi _{\ell }(x)=0$ whenever $x_1\in {\mathbb{R}}\setminus (\alpha _{4\ell +1}, \alpha _{4\ell -2})$ . Hence, setting $\psi (x)=\sum _{k=1}^{K/4}\psi _{\ell }(x)$ yields the desired network.

Proof of Theorem 2.2. We begin by defining the sets $\mathcal{C}_1$ and $\mathcal{C}_2$ . Let $\mathcal{C}_1 = \{ f_a: {\mathbb{R}}^d \to [0,1] \, \vert \, a \in [1/2,1]\}$ , where $f_a$ is defined as in (5.2). Since all norms on finite dimensional vector spaces are equivalent, let $D \gt 0$ be such that $\|\cdot \| \leq D \|\cdot \|_1$ . To define the set of distributions, we first set $\delta =\epsilon /(2D)$ . For each $\kappa \in [1/4,3/4]$ , define the distribution $\mathcal{D}_\kappa$ on $[0,1]^{N_0}$

\begin{align*} X\sim \mathcal{D_{\kappa } }\quad \iff \quad {\mathbb{P}}(X=x)=\begin{cases} \mathrm{p}_k & \text{if }x= x^{k,\delta }\\ 0 &\text{otherwise} \end{cases}. \end{align*}

where $\mathrm{p}_{2j-1}=\mathrm{p}_{2j}=C_\zeta (3/2)j^{-3/2}$ for $j\in \mathbb{N}$ and $x^{k,\delta }$ is defined according to (5.1). We set $\mathcal{C}_2 = \{\mathcal{D}_\kappa \, \vert \, \kappa \in [1/4,3/4]\}$ .

Let $c_1$ , $c_2$ and $C_\zeta (3/2)$ be the constants defined in Lemma5.6 with $\gamma$ set to $3/2$ . We choose the constant $C$ so that each of the following hold:

(5.11) \begin{align} C& \geq 4^3c_1^{-6}, \end{align}

(5.12) \begin{align} C &\geq 200 \log (8)^{3/2} c_1^{-3/2}, \text{ and} \end{align}

(5.13) \begin{align} C &\geq 4\cdot (8c_2)^{2}. \end{align}

Fix $a \in [1/2,1]$ so that $f_a \in \mathcal{C}_1$ and $\kappa \in [1/4,3/4]$ so that $\mathcal{D}_{\kappa } \in \mathcal{C}_2$ . Let $\mathcal{T} = \{x^1, \ldots , x^r\}$ and $\mathcal{V} = \{y^1, \ldots , y^s\}$ be the random multisets drawn from this distribution as in the statement of the theorem. Then by the definition of the distribution $\mathcal{D}_\kappa$ , we can write (after removing repetitions and reordering) $\mathcal{T} \cup \mathcal{V}$ as $ S\,:\!=\,\mathcal{T} \cup \mathcal{V} = \{ x^{Z_1,\delta },x^{Z_2,\delta },x^{Z_3,\delta }, \dotsc ,x^{Z_N,\delta }\}$ where the random variable $N$ satisfying $N \leq r+s$ is the number of unique elements in $\mathcal{T}\cup \mathcal{V}$ and where $Z_1 \lt Z_2 \lt \dotsb \lt Z_{N}$ . For shorthand, we also set $z^j = x^{Z_j,0}$ for $j=1,2,\dotsc ,N$ .

Since $C/2 \geq 2 \cdot (8c_2)^2$ (by (5.13)) and $C(r\vee s)^2/(2p^2) \geq 4^3c_1^{-6}/2 \geq 2$ (by (5.11) and the facts that $(r \vee s)/p \geq 1$ and $c_1^{-1} \geq 1$ ) we obtain

\begin{align*} \frac {C(r\vee s)^2}{p^2} = \frac {C(r\vee s)^2}{2p^2} + \frac {C(r\vee s)^2}{2p^2} \geq 2\cdot \frac {(8c_2)^2(r\vee s)^2}{p^2} +2 \geq 2 \left \lceil {\left (\frac {8c_2 (r \vee s)}{p}\right )^2+1}\right \rceil - 2 \end{align*}

and thus item (ii) of Lemma5.6 with $\gamma = 3/2$ yields

(5.14) \begin{align} &{\mathbb{P}}\left (\max \{k\in \mathbb{N}\,\vert \, x^{k,\delta }\in \mathcal{T} \cup \mathcal{V} \} \leq \left \lceil {\frac {C (r\vee s)^2}{\mathrm{p}^2}} \right \rceil \right )={\mathbb{P}}\left (Z_N \leq \left \lceil {\frac {C (r\vee s)^2}{\mathrm{p}^2}} \right \rceil \right ) \notag \\ &\quad\geq {\mathbb{P}}\left (Z_N \leq 2{\left \lceil \left (\frac {8c_2 (r \vee s)}{p}\right )^2+1\right \rceil } - 2 \right )\notag \geq 1- \frac {c_2 (r+s)}{\left\lfloor{\left\lceil\left(\frac{8c_2 (r \vee s)}{p}\right)^2+1\right\rceil} - 1 \right\rfloor} \notag \\ &\quad\geq 1- \frac {c_2 (r+s)}{ (8 c_2 (r \vee s)/\mathrm{p})} \geq 1-\mathrm{p}/4. \end{align}

Writing $N_{\text{prod}}\,:\!=\,(N_1+1)\cdots (N_{L-1}+1)$ , by the Assumptions (2.3) and (5.12), we obtain

\begin{align*} \lfloor c_1 (r+s)^{2/3} \rfloor \geq \lfloor C^{2/3}c_1 qN_{\text{prod}} \rfloor \geq \lfloor 200^{2/3} \log (8)qN_{\text{prod}} \rfloor \geq 30 qN_{\text{prod}}. \end{align*}

Therefore, we can apply item (iii) of Lemma5.6 to see that

(5.15) \begin{align} &{\mathbb{P}}\Big (\sum _{i=1}^{N-1} \chi _{\{ f_a(z^{i+1}) \neq f_a(z^{i}) \}} \gt 6 qN_{\text{prod}} \Big ) = {\mathbb{P}}\Big (\sum _{i=1}^{N-1} \chi _{\{ Z_{i+1} - Z_i \text{ odd } \}} \gt 6 qN_{\text{prod}} \Big )\notag \\&\quad\geq \sum _{n=\lfloor c_1(r+s)^{\frac {2}{3}}\! \rfloor }^{r+s} \!\!\!\!\!\!{\mathbb{P}}\Big ( \sum _{i=1}^{n-1} \chi _{\{ Z_{i+1} - Z_i \text{ odd } \}}\gt \frac {n}{5} \,\Big \rvert \, N = n\Big ){\mathbb{P}}(N=n)\notag \\ &\quad\geq \sum _{n=\lfloor c_1(r+s)^{\frac {2}{3}}\! \rfloor }^{r+s} \!\!\!\!\!\!\exp \left (-\frac {n}{100}\right ){\mathbb{P}}(N=n)\notag \\ &\quad\geq \left [1- \exp \left (-\left \lfloor {\frac {c_1 (r+s)^{\frac {2}{3}}} {100}}\right \rfloor \right ) \right ]\cdot {\mathbb{P}}( N \geq \lfloor c_1 (r+s)^{2/3} \rfloor ) \end{align}

where the application of Lemma5.6 is justified by the bound $\lfloor c_1 (r+s)^{2/3} \rfloor \geq 30 qN_{\text{prod}} \geq 10$ and the initial equality in the first line is justified by the fact that $f_a(z^i)$ depends only on the parity of $i$ , a fact itself readily seen from the definition of $f_a$ and $z^i$ .

Now, by differentiating it is easy to see that the function $p \mapsto p\log (8/p)$ is increasing on $(0,1)$ . Hence for $p \lt 1$ , we have $p^{-2}\log (8) \gt p^{-1} \log (8)\gt \log (8/p)$ and so combining this with (2.3) and (5.12) gives

(5.16) \begin{equation} {\left \lfloor \frac {c_1 (r+s)^{2/3}}{100}\right \rfloor } \geq {\left \lfloor \frac {c_1 C^{2/3}\mathrm{p}^{-2}}{100}\right \rfloor } \geq {\left \lfloor \frac {200^{2/3} \mathrm{p}^{-2}\log (8)}{100}\right \rfloor }\geq \mathrm{p}^{-2}\log (8) - 1 \geq \log (8/\mathrm{p}) - 1. \end{equation}

Furthermore, using item (i) of Lemma5.6 with $\gamma =3/2$ , we obtain ${\mathbb{P}}\big (N \geq c_1 (r+s)^{2/3}\big )\geq 1- c_1^{-2}(r+s)^{-1/3}\geq 1-\mathrm{p}/4,$ where the final bound follows because $r+s \geq Cp^{-3}$ (which, in turn, is due to the Assumption (2.3)) and (5.11). Using this result together with (5.16) in (5.15) yields

(5.17) \begin{equation} {\mathbb{P}}\Big (\sum _{i=1}^{N-1} \chi _{\{ f_a(z^{j+1}) \neq f_a(z^{j}) \}} \gt 6 qN_{\text{prod}} \Big ) \gt \left ( 1- e\mathrm{p}/8\right )\left (1-\mathrm{p}/{4}\right ) \gt 1- \mathrm{p}/2 \end{equation}

Combining (5.14) and (5.17), we see that the probability that both

(5.18) \begin{equation} \max \{k\in \mathbb{N}\,\vert \, x^{k,\delta }\in \mathcal{T}\cup \mathcal{V}\} \leq {\left \lceil \frac {C(r\vee s)^2}{\mathrm{p}^2} \right \rceil } \quad \text{and } \sum _{i=1}^{N-1} \chi _{\{ f_a(z^{i+1}) \neq f_a(z^{i}) \}} \gt 6 qN_{\text{prod}} \end{equation}

occur is at least $1-(\mathrm{p}/4+ \mathrm{p}/2)\gt 1- \mathrm{p}$ . We will now proceed to show that each of (i) through (iii) listed as in the statement of Theorem 2.2 hold assuming that this event occurs.

To see that $\mathcal{T}, \mathcal{V} \in \mathcal{S}^f_{\varepsilon ((r\vee s)/\mathrm{p})}$ , note that (5.12) and $c_1^{-1} \geq 1$ yields $C^{2}t^{2}\geq (4\lceil t \rceil +3)(4\lceil t \rceil +4)$ for all $t\geq 1$ . Applying this inequality with $t = C ((r\vee s)/\mathrm{p})^2\geq 1$ , we deduce that

(5.19) \begin{align} \varepsilon \left [\frac {C(r\vee s)}{\mathrm{p}}\right ] = C^{-2} \left ( \frac {C(r\vee s)^2}{\mathrm{p}^2}\right )^{-2}&\leq \left [\left (4 {\left \lceil \frac {C(r\vee s)^2}{\mathrm{p}^2} \right \rceil } +3\right )\left (4 {\left \lceil \frac {C(r\vee s)^2}{\mathrm{p}^2}\right \rceil } +4\right )\right ]^{-1}\notag \\&=\varepsilon '\left ({\left \lceil \frac {C(r\vee s)^2}{\mathrm{p}^2}\right \rceil }\right ) , \end{align}

where $\varepsilon '(n)=[(4n+3)(4n+4)]^{-1}$ . Therefore because we assume that $\max \{k\in \mathbb{N}\,\vert \, x^{k,\delta }\in \mathcal{T}\cup \mathcal{V} \} \leq {\left \lceil \frac {C(r\vee s)^2}{\mathrm{p}^2} \right \rceil }$ , Lemma5.2 yields $\mathcal{T},\mathcal{V}\subset \{x^{1,\delta }, \ldots , x^{ \lceil {C(r\vee s)^2/\mathrm{p}^2} ,\delta \rceil } \}\in \mathcal{S}^{f_a}_{\varepsilon '(\lceil {C(r\vee s)^2/\mathrm{p}^2}\rceil )}\subset \mathcal{S}^{f_a}_{\varepsilon (C(r\vee s)/\mathrm{p})}$ .

The construction of $\phi$ satisfying (2.5) is immediate: we take $\phi$ to be the neural network $\tilde {\varphi }$ defined in Lemma5.3. We conclude that ${\phi }(x) = f_a(x)$ for all $x \in \mathcal{T} \cup \mathcal{V}$ (this establishes (2.5)). Because ${\phi }(x) = f_a(x)$ for all $x \in \mathcal{T}$ and because $\mathcal{R} \in \mathcal{CF}_{r}$ we conclude that $\mathcal{R} \left (\{\phi ({x}^j)\}_{j=1}^r,\{f({x}^j)\}_{j=1}^r\right ) = 0$ . Thus (2.4) holds, completing the proof of (i).

Our next task will be to show that if $\hat \phi \in \mathcal{NN}_{\mathbf{N},\kern0.3pt L}$ and $g: {\mathbb{R}} \to {\mathbb{R}}$ is monotonic, then there is a subset $\mathcal{\tilde T}\subset \mathcal{T} \cup \mathcal{V}$ of the combined training and validation set of size $|\mathcal{\tilde T}| \geq q$ , such that there exist uncountably many universal adversarial perturbations $\eta \in \mathbb{R}^d$ so that for each $x \in \mathcal{\tilde T}$ Eq. (2.6) applies.

To this end, note that (5.18) implies that there exist natural numbers $k_1\lt k_2\lt \ldots \lt k_{6qN_{\text{prod}}}$ such that $z^{k_i}_1\gt z^{k_{i+1}}_1$ and $f_a(z^{k_i})\neq f_a(z^{k_{i+1}})$ for all $i\in \{1,\dotsc , 6qN_{\text{prod}}-1\}$ . Moreover, by the definition of $\mathcal{T}$ , $\mathcal{V}$ and $S$ , there exist $m_i$ such that $z^{k_i}_1 = x^{m_i,\delta }_1$ and such that $x^{m_i,\delta } \in \mathcal{T} \cup \mathcal{V}$ . For such $i$ and any $\omega \in [0,\delta \wedge \varepsilon ((r\vee s)/\mathrm{p}) )$ , we define the vectors $w^{i,\omega } = z^{k_i}+\omega e_1$ . We also define the sets $\mathcal{W}^{\omega }\,:\!=\,\{ w^{i,\omega } \, \vert \, i\in \{1,\dotsc , 6qN_{\text{prod}}\}\}$ .

Because of the definition of $x^{k,0}$ given in (5.1) and the definition of $z^{k_i}$ , we have $z^{k_i}_2 = z^{k_i}_3 = \dotsb = z^{k_i}_d = 0$ and $z^{k_i} = x^{m_i,0}$ . In particular, $\{z^{k_i} \, \vert \, i \in \{1,\dotsc , 6qN_{\text{prod}}\}\} = \{ x^{m_i,0} \, \vert \, i\in \{1,\dotsc , 6qN_{\text{prod}}\} \} \in \mathcal{S}^{f_a}_{\varepsilon ((r\vee s)/\mathrm{p})}$ where we have used Lemma5.2 and the bound (5.19). Since $\|z ^{k_i} - w^{i,\omega }\|_{\infty } = \omega \lt \varepsilon ((r\vee s)/\mathrm{p})$ , we conclude that $f_a(z^{k_i}) = f_a(w^{i,\omega })$ for $i \in \{1,\dotsc , 6qN_{\text{prod}}\}$ . Thus, $f_a(w^{i,\omega }) = f_a(z^{k_i}) \neq f_a(z^{k_{i+1}}) = f_a(w^{i+1,\omega })$ for $i \in \{1,\dotsc , 6qN_{\text{prod}}-1\}$ .

We can now use Lemma5.4 to conclude that for each $\omega \in [0,\delta \wedge \varepsilon ((r\vee s)/\mathrm{p}) )$ there exists a set $\mathcal{I}^{\omega }$ and a set $\mathcal{U}^{\omega }\subset \mathcal{W}^{\omega }$ with the following properties:

1. (1) $\mathcal{I}^{\omega } \subset \{1,2,\dotsc ,6qN_{\text{prod}}\}$

2. (2) $\mathcal{U}^{\omega } = \{ w^{i,\omega } \, \vert \, i\in \mathcal{I}^{\omega }\}$

3. (3) For all $w \in \mathcal{U}^{\omega }$ , $|g(\hat \phi (w)) - f_a(w)| \geq 1/2$ .

4. (4) $|\mathcal{U}^{\omega }| \geq 2q$ .

By the pigeonhole principle and the finiteness of $\{1,2,\dotsc ,6qN_{\text{prod}}\}$ , there exists an uncountable set $\Omega \subset [0,\delta \wedge \varepsilon ((r\vee s)/\mathrm{p}))$ such that for all $\omega \in \Omega$ , $\mathcal{I}^{\omega }$ is independent of $\omega$ . Let $\mathcal{I}$ denote this common value and let $\mathcal{I}_{E}\,:\!=\,\{i \, \vert \, i \in \mathcal{I}, m_i \text{ even}\}$ and $\mathcal{I}_{O}\,:\!=\,\{i \, \vert \, i \in \mathcal{I}, m_i \text{ odd}\}$ . Note that $|\mathcal{I}| \geq 2q$ ; otherwise, $|\mathcal{U}^{\omega }| \lt 2q$ for some $\omega$ . Therefore, at least one of $|\mathcal{I}_E|\geq q$ or $|\mathcal{I}_O|\geq q$ : we now split into two cases depending on which of these two sets has cardinality at least $q$ .

Case 1: $|\mathcal{I}_E|\geq q$ .

In this case, we choose $\tilde {\mathcal{T}} = \{x^{m_i,\delta } \, \vert \, i \in \mathcal{I}_E\}$ . For each $\omega \in \Omega$ , define $\eta ^{\omega } = (\omega ,-\delta ,0,\dotsc ,0) \in {\mathbb{R}}^d$ and $\mathcal{H} = \{ \eta ^{\omega } \, \vert \, \omega \in \Omega \}$ . Then the set $\mathcal{H}$ is uncountable, for each $i \in \mathcal{I}_E$ and $\omega \in \Omega$ we have $x^{m_i,\delta } + \eta ^{\omega } = w^{i,\omega }$ , $|g(\hat \phi (x^{m_i,\delta } + \eta ^{\omega })) - f_a(x^{m_i,\delta } + \eta ^{\omega })| = |g(\hat \phi (w^{i,\omega })) - f_a(w^{i,\omega })| \geq 1/2$ and $\|\eta \| \leq D\|\eta ^{\omega }\|_1 = D(\omega + \delta ) \leq 2D\delta \leq \epsilon$ . Furthermore, $|\text{supp}(\eta ^\omega )| = 2$ . We conclude that (2.6) holds.

Case 2: $|\mathcal{I}_O| \geq q$

In this case, we choose $\tilde {\mathcal{T}} = \{x^{m_i,\delta } \, \vert \, i \in \mathcal{I}_O\}$ . For each $\omega \in \Omega$ , define $\eta ^{\omega } = (\omega ,0,0,\dotsc ,0) \in {\mathbb{R}}^d$ and $\mathcal{H} = \{ \eta ^{\omega } \, \vert \, \omega \in \Omega \}$ . Then the set $\mathcal{H}$ is uncountable, for each $i \in \mathcal{I}_O$ and $\omega \in \Omega$ we have $x^{m_i,\delta } + \eta ^{\omega } = w^{i,\omega }$ , $|g(\hat \phi (x^{m_i,\delta } + \eta ^{\omega })) - f_a(x^{m_i,\delta } + \eta ^{\omega })| = |g(\hat \phi (w^{i,\omega })) - f_a(w^{i,\omega })| \geq 1/2$ and $\|\eta ^{\omega }\| \leq D\|\eta ^{\omega }\|_1 = D\omega \leq D\delta \leq \epsilon$ . Furthermore, $|\text{supp}(\eta ^\omega )| = 1$ . We conclude that (2.6) holds.

Finally, we must show the existence of $\psi$ , which we do with the help of Lemma5.7. To this end, we set $K= \lceil C ((r\vee s)/\mathrm{p})^2 \rceil$ and define $\{\alpha _j\}_{j=1}^{2K}$ by $ \alpha _{2k-1}=x_1^{k,\delta }+ \varepsilon ((r\vee s)/\mathrm{p})$ , $ \alpha _{2k}= x_1^{k,\delta }-\varepsilon ((r\vee s)/\mathrm{p})$ for $k=1,\ldots , K$ . We first claim that $0 \lt \alpha _{2K} \lt \alpha _{2K-1} \lt \dotsb \lt \alpha _2 \lt \alpha _1 \lt 1$ .

Because $C \geq 4^3$ , $p \leq 1$ and $(r \vee s) \geq 1$ we have

\begin{align*} \alpha _1 = \frac {a}{2-\kappa } + \frac {\mathrm{p}^4}{C^4 (r\vee s)^4} \leq \frac {1}{(2-3/4)} + \frac {1}{C^4} \lt 1 \end{align*}

and similarly we obtain $2\lceil C ((r\vee s)/\mathrm{p})^2 \rceil + 1 - \kappa \leq 2(C ((r\vee s)/\mathrm{p})^2) + 2 - \kappa \leq 4(C ((r\vee s)/\mathrm{p})^2)$ . Therefore,

\begin{align*} \alpha _{2K} = \frac {a}{2\lceil C ((r\vee s)/\mathrm{p})^2 \rceil + 1 - \kappa } - \frac {\mathrm{p}^4}{C^4 (r\vee s)^4} &\geq \frac {a}{4C ((r\vee s)/\mathrm{p})^2} - \frac {\mathrm{p}^4}{C^4 (r\vee s)^4} \\&\geq \frac {\mathrm{p}^2}{8C (r\vee s)^2} - \frac {\mathrm{p}^2}{4^{12}C (r\vee s)^2}\gt 0 \end{align*}

A simple calculation also shows that for each $j=1,\ldots , K -1$

\begin{align*} x^{j,\delta }_1 - x^{j+1,\delta }_1 = \frac {a}{(j+2-\kappa )(j+1-\kappa )} \geq \frac {a}{(K+1-\kappa )(K-\kappa )} \geq [2(K+1-\kappa )(K-\kappa )]^{-1}. \end{align*}

On the other hand, once again employing the result that $C^{2}t^{2}\geq (4\lceil t \rceil +3)(4\lceil t \rceil +4)$ , for all $t\geq 1$ , (which is a consequence of (5.12)) with $t = C((r\vee s)/\mathrm{p})^2$ we obtain

\begin{align*} 2\varepsilon ((r\vee s)/\mathrm{p}) = 2 C^{-2} ( C((r\vee s)/\mathrm{p})^2)^{-2}\leq 2[(4K+3)(4K+4)]^{-1} \lt [2(K+1 - \kappa )(K -\kappa )]^{-1}. \end{align*}

We therefore conclude that $\alpha _{2j-1} \gt \alpha _{2j} = x^{j,\delta }_1 -\varepsilon ((r\vee s)/\mathrm{p}) \gt x^{j+1,\delta }_1 +\varepsilon ((r\vee s)/\mathrm{p}) = \alpha _{2j+1}$ , and thus the conditions to apply Lemma5.7 are met.

Now, let $\psi$ be the network provided by Lemma5.7 with this sequence $\{\alpha _j\}_{j=1}^{2K}$ . Because of the definition of $\alpha _j$ and the conclusion of Lemma5.7 we have

\begin{align*} \psi (x) = \begin{cases} 0 & \text{ if } x_1 \in [x^{k,\delta }_1 - \varepsilon ((r\vee s)/\mathrm{p}), x^{k,\delta }_1 + \varepsilon ((r\vee s)/\mathrm{p})] \text{ and } k \text{ is odd} \\ 1 & \text{ if } x_1 \in [x^{k,\delta }_1 - \varepsilon ((r\vee s)/\mathrm{p}), x^{k,\delta }_1 + \varepsilon ((r\vee s)/\mathrm{p})] \text{ and } k \text{ is even} \end{cases} \end{align*}

Moreover, because of Lemma5.2, the fact that the value of $f_a(x)$ depends only on $x_1$ and the bound (5.19)

\begin{align*} f_a(x) = \begin{cases} 0 & \text{ if } x_1 \in [x^{k,\delta }_1 - \varepsilon ((r\vee s)/\mathrm{p}), x^{k,\delta }_1 + \varepsilon ((r\vee s)/\mathrm{p})] \text{ and } k \text{ is odd} \\ 1 & \text{ if } x_1 \in [x^{k,\delta }_1 - \varepsilon ((r\vee s)/\mathrm{p}), x^{k,\delta }_1 + \varepsilon ((r\vee s)/\mathrm{p})] \text{ and } k \text{ is even} \end{cases} \end{align*}

In particular, $\psi (x) = f_a(x)$ whenever $x_1 \in [x^{k,\delta }_1 - \varepsilon ((r\vee s)/\mathrm{p}), x^{k,\delta }_1 + \varepsilon ((r\vee s)/\mathrm{p})]$ for any $k \in \{1,2,\dotsc ,K\}$ .

To see that $\psi (x) = f_a(x)$ for all $x \in \mathcal{B}_{\varepsilon ((r\vee s)/\mathrm{p})}^{\infty }(\mathcal{T} \cup \mathcal{V})$ , note that, for every $x \in \mathcal{B}_{\varepsilon ((r\vee s)/\mathrm{p})}^{\infty }(\mathcal{T} \cup \mathcal{V})$ , there exists an $x^{k,\delta }\in \mathcal{T}\cup \mathcal{V}$ such that $\|x^{k,\delta }-x\|_\infty \leq \varepsilon ((r\vee s)/\mathrm{p})$ . Then, by the assumption that $\max \{\ell \in \mathbb{N}\,\vert \, x^{\ell ,\delta }\in \mathcal{T} \cup \mathcal{V} \} \leq \lceil C ((r \vee s)/\mathrm{p})^2 \rceil$ occurs in (5.18), we have $k\leq K$ , and so $x_1 \in [x^{k,\delta }_1 - \varepsilon ((r\vee s)/\mathrm{p}), x^{k,\delta }_1 + \varepsilon ((r\vee s)/\mathrm{p})]$ . But we have already shown that for such $x$ , $\psi (x) = f_a(x)$ . Thus, the proof of the theorem is complete.

Proof. By Lemma5.3, there exists a neural network $\tilde \varphi \in \mathcal{NN}_{\mathbf{N},\kern0.3pt L}$ with $\tilde \varphi (x^{k,\delta }) = f_a(x^{k,\delta })$ for all $k$ . In particular, $ \mathcal{R} \left (\{\tilde \varphi ({x}^{j,\delta })\}_{j=1}^r,\{f({x}^{j,\delta })\}_{j=1}^r\right ) = 0$ . Thus, by (5.24) and the definition of the approximate argmin as in (3.3), we must have that

\begin{align*} \mathcal{R} \left (\{ \phi ({x}^{j,\delta })\}_{j=1}^r,\{f({x}^{j,\delta })\}_{j=1}^r\right ) \leq \epsilon \end{align*}

and the conclusion of the claim follows because $\mathcal{R}\in \mathcal{CF}^{\epsilon ,\hat \epsilon }_{r}$ as defined in (3.4).

Proof of Proposition 5.26. As in the proof of Theorem 2.2, we begin by defining the sets $\mathcal{C}_1$ and $\mathcal{C}_3$ . Let $\mathcal{C}_1 = \{ f_a: {\mathbb{R}}^d \to [0,1] \, \vert \, a \in [1/2,1]\}$ , where $f_a$ is defined as in (5.2). Fix $a \in [1/2,1]$ and $\kappa \in [1/4,3/4]$ , define $\mathcal{T}^{\kappa }_{\delta }\,:\!=\,\{x^{1,\delta },x^{2,\delta },\dotsc ,x^{r,\delta }\}$ where the values $x^{i,\delta }$ (each depending on $\kappa$ and $a$ ) are defined in (5.1). We define $\hat \Omega ^{\kappa }\,:\!=\,\{\mathcal{T}^{\kappa }_{\delta } \, \vert \, \delta \in [0,\epsilon '(r))$ . By Lemma5.2, we have $\mathcal{T}^{\kappa }_{\delta } \in \mathcal{S}^{f_a}_{\varepsilon '(r)}$ so that $\hat \Omega ^{\kappa } \subset \Omega ^{\mathcal{NN}}$ . Note also that noting that the $\hat \Omega ^{\kappa }$ are disjoint as an immediate consequence of (5.1). Finally, we set $\mathcal{C}_3\,:\!=\,\{ \hat \Omega ^{\kappa } \, \vert \, \kappa \in [1/4,3/4]\}$ , .

Now we have defined $\mathcal{C}_1$ and $\mathcal{C}_3$ , we will show that for any $\kappa \in [1/4,3/4]$ the computational problem

\begin{align*} \{\Xi ^{\mathcal{NN}},\hat \Omega ^\kappa ,\mathcal{M}^{\mathcal{NN}},\Lambda ^{\mathcal{NN}}\}^{\Delta _1} \end{align*}

has breakdown epsilon $\epsilon _{\mathbb{P}h\mathrm{B}}^{\mathrm{s}}(\mathrm{p}) \geq 1/4 - \hat {\epsilon }/2$ , for all $\mathrm{p} \in [0,1/2)$ . This will be done using Proposition 5.25. We will define $\iota ^0\,:\!=\,\mathcal{T}^{\kappa }_{0}$ and $\iota ^1_n\,:\!=\,\mathcal{T}^{\kappa }_{4^{-n}}$ .

By (5.1), we see that $\|x^{\,j,4^{-n}}-x^{j,0}\|_\infty \leq 4^{-n}$ for $j=1,2,\dotsc ,r$ . Hence (recalling the definition of $\Lambda ^{\mathcal{NN}}$ ), property (Pa) from Proposition 5.25 holds.

Fix $n \in \mathbb{N}$ sufficiently large and let $\phi _0$ and $\phi _n$ be arbitrary neural networks so that

(5.25) \begin{equation} \begin{split} &\phi _0 \in \underset {\varphi \in \mathcal{NN}_{\mathbf{N},\kern0.3pt L}}{\operatorname {argmin}_{\epsilon }} \left (\{\varphi (x^{j,0})\}_{j=1}^r ,\{f_a(x^{j,0})\}_{j=1}^r\right )\\ &\phi _n \in \underset {\varphi \in \mathcal{NN}_{\mathbf{N},\kern0.3pt L}}{\operatorname {argmin}_{\epsilon }} \left (\{\varphi (x^{j,4^{-n}})\}_{j=1}^r ,\{f_a(x^{j,4^{-n}})\}_{j=1}^r\right ). \end{split} \end{equation}

By Lemma5.4 and the assumption that $|\mathcal{T}^{\kappa }_0| = r \geq 3(N_1+1) \dotsb (N_{L-1}+1)$ , we conclude that

\begin{align*} \max \limits _{j=1,2,\dotsc ,r} |\phi _0(x^{j,0}) - f_a(x^{j,0})| \geq 1/2. \end{align*}

By contrast, Lemma5.27 shows that $\max _{j=1,2,\dotsc ,r} |\phi _n(x^{j,4^{-n}}) - f_a(x^{j,4^{-n}})| \leq \hat \epsilon$ . Combining these two results and the fact that $f_a(x^{j,0}) = f_a(x^{j,4^{-n}})$ for each $j=1,2,\dotsc ,r$ yields

\begin{align*} \max _{j=1,2,\dotsc ,r} |\phi _0(x^{j,0}) - \phi _n(x^{j,4^{-n}})| \geq 1/2 - \hat \epsilon . \end{align*}

Therefore, since both the $\ell ^1$ and $\ell ^2$ norms are bounded from below by the $\ell ^{\infty }$ norm and $\phi _0$ and $\phi _n$ were chosen arbitrarily according to (5.25), we have $ \inf _{\upsilon ^1 \in \Xi (\iota ^1_n), \upsilon ^2 \in \Xi (\iota ^0)}d_{\mathcal{M}}(\upsilon ^1,\upsilon ^2) \geq 1/2 - \hat \epsilon$ where $d_{\mathcal{M}}$ is the $\ell ^{*}$ norm with $*= 1,2$ or $\infty$ . Hence, property (Pb) from Proposition 5.25 holds with $\kappa = 1/2 - \hat \epsilon$ , thereby concluding the proof.