Appendix A. Proofs of Proposition 2 and Theorem 1

We present the following lemma, which is used throughout our work.

Lemma 1. Under Assumption 1,

\begin{align*} \mathbb{E} \left(\nabla l(\theta,u)\right)=\nabla g(\theta). \end{align*}

Proof. We prove this for one dimension as that suffices for the general case as expectation distributes over all the components of the vector. Here $\theta$ is a fixed point at which we differentiate. Note that

\begin{align*} \frac{l(\theta_1,u)-l(\theta,u)}{\theta_1-\theta}=\nabla l(\xi,u) \end{align*}

for some $\xi$ using the mean value theorem. Note that, since differentiation is a local property, we can force $\theta_1 \in B(\theta,1)$ , where $B(\theta,1)$ denotes the ball centered at $\theta$ with radius 1. This also forces $\xi \in B(\theta,1)$ . In addition, note,

\begin{align*} \left|\nabla l(\xi,u)\right|&\le \left|\nabla l(\theta,u)\right|+h_1(u)\left|\xi-\theta\right|\\ & \le \left|\nabla l(\theta,u)\right|+h_1(u). \end{align*}

The last line follows as $\xi \in B(\theta,1)$ . This implies

\begin{align*} \frac{l(\theta_1,u)-l(\theta,u)}{\theta_1-\theta} & \le \left|\frac{l(\theta_1,u)-l(\theta,u)}{\theta_1-\theta}\right|\\ &\le \left|\nabla l(\theta,u)\right|+h_1(u). \end{align*}

The last term is independent of $\theta_1$ and is integrable. Hence, we can use DCT and we are done.

Proof of Proposition 2. We begin by noting the fact that

\begin{align*} \mathbb{E}\left|\sqrt{m}\left(\sum_{i=1}^{n}w_i \nabla l(\theta,u_i) -\nabla g(\theta)\right)-\sqrt{n}\left(\sum_{i=1}^{n}\frac{1}{n}\nabla l(\theta,u_i) -\nabla g(\theta)\right)\right|^2 & \\ \quad=\mathbb{E}\left|\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\nabla l(\theta,u_i)+\left(\sqrt{n}-\sqrt{m}\right)\nabla g(\theta)\right|^2. \end{align*}

Now, the last term is equal to

\begin{align*} &\mathbb{E}\left\langle\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\nabla l(\theta,u_i),\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\nabla l(\theta,u_i)\right\rangle\\ &+ 2 \ \mathbb{E}\left\langle\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\nabla l(\theta,u_i),\left(\sqrt{n}-\sqrt{m}\right)\nabla g(\theta)\right\rangle + \left(\sqrt{n}-\sqrt{m}\right)^2\left|\nabla g(\theta)\right|^2. \end{align*}

We condition on $w=(w_1,w_2,\ldots,w_n)$ and get the above expression equal to

\begin{align*} & \mathbb{E}\left[\sum_{1\le i,j \le n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\left(\sqrt{m}w_j-\frac{1}{\sqrt{n}}\right)\mathbb{E}_w\left\langle\nabla l(\theta,u_i),\nabla l(\theta,u_j)\right\rangle\right] \\ &+ 2 \ \mathbb{E}\left[\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\left(\sqrt{n}-\sqrt{m}\right)\left|\nabla g(\theta)\right|^2)\right]\\ &+\left(\sqrt{n}-\sqrt{m}\right)^2 \ \left|\nabla g(\theta)\right|^2. \end{align*}

Here $\mathbb{E}_w$ denotes the conditional expectation with respect to the weights. Using the fact $\mathbb{E}(w_i)=1/n$ and some minor manipulation, the second term is $-2\left(\sqrt{n}-\sqrt{m}\right)^2\left|\nabla g(\theta)\right|^2$ . Hence,

\begin{align*} & \mathbb{E}\left[\sum_{1\le i,j \le n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\left(\sqrt{m}w_j-\frac{1}{\sqrt{n}}\right)\mathbb{E}_w\left\langle\nabla l(\theta,u_i),\nabla l(\theta,u_j)\right\rangle\right]\\ &\quad -\left(\sqrt{n}-\sqrt{m}\right)^2 \left|\nabla g(\theta)\right|^2\\ &= \mathbb{E}\Bigg[\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)^2 \mathbb{E}_w \left|\nabla l(\theta,u_i)\right|^2\\ &\quad +\sum_{1\le i,j \le n, i\ne j} \left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\left(\sqrt{m}w_j-\frac{1}{\sqrt{n}}\right)\mathbb{E}_w\left(\nabla l(\theta,u_i)^{\mathsf{T}}\nabla l(\theta,u_j)\right)\Bigg]\\ & \quad -\left(\sqrt{n}-\sqrt{m}\right)^2 \left|\nabla g(\theta)\right|^2. \end{align*}

The final term equals

\begin{align*} &\mathbb{E}\Bigg[\sum_{i=1}^{n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)^2 \left(Tr \sigma^2(\theta)+\left|\nabla g(\theta)\right|^2\right)\\ &\quad +\sum_{1\le i,j \le n}\left(\sqrt{m}w_i-\frac{1}{\sqrt{n}}\right)\left(\sqrt{m}w_j-\frac{1}{\sqrt{n}}\right) \left|\nabla g(\theta)\right|^2\\ & \quad -\left(\sqrt{n}-\sqrt{m}\right)^2 \left|\nabla g(\theta)\right|^2\Bigg]. \end{align*}

Using the covariance structure of the weights, the last expression is reduced to

\begin{align*} & 2\left[\textrm{Tr} \sigma^2(\theta)+\left|\nabla g(\theta)\right|^2\right]\left(1-\sqrt{\frac{m}{n}}\right)+\left|\nabla g(\theta)\right|^2\left[(\sqrt{m}-\sqrt{n})^2-2+2\sqrt{\frac{m}{n}}\right]\\ &\quad -\left(\sqrt{n}-\sqrt{m}\right)^2\left|\nabla g(\theta)\right|^2\\ &\quad =2\left(1-\sqrt{\frac{m}{n}}\right)\textrm{Tr} \sigma^2(\theta). \end{align*}

Using this, in the regime $m/n\to 1$ as $n\to \infty$ , we get the first conclusion for Proposition 2.

The second conclusion to Proposition 2 can be derived similarly.

Lemma 2. Let Assumption 5 hold. Then, we have

\[m\sum_{i=1}^{n} w_i^2 \xrightarrow[L^1]{\textrm{a.s.}} 1,\]

when $m/n\to \gamma^*$ and $0\le \gamma^*\le 1$ .

Proof. Noting the fact that $w=\Sigma^{1/2}X+1/n\,\tilde{1}$ , we have

\[m\sum_{i=1}^{n} w_i^2=m\left(X^{\mathsf{T}}\Sigma X \ +\frac{1}{n}\right).\]

We also note that

\[\Sigma=\frac{n-m}{mn(n-1)}\left(I-\frac{1}{n}\tilde{1}\tilde{1}^{\mathsf{T}}\right).\]

This leads to that

\begin{align*} m\sum_{i=1}^{n} w_i^2 =& \frac{n-m}{n(n-1)}X^{\mathsf{T}}\left(I-\frac{1}{n}\tilde{1}\tilde{1}^{\mathsf{T}}\right)X+\frac{m}{n}\\ =& \frac{n-m}{n-1}\left(\frac{1}{n}X^{\mathsf{T}}X-\bar{X}^2\right)+\frac{m}{n}. \end{align*}

Now, the above expression converges to 1 both almost surely and in $L_1$ -norm, whatever $\gamma^*$ is. If $\gamma^*=0$ , the second term converges to 0 and the first term converges to 1 almost surely using the law of large numbers and also $L_1$ . If $\gamma^*=1$ , the second term converges to 1 and the first term converges to 0. If $0<\gamma^*<1$ , we have the second term converging to $\gamma^*$ and the first converging to $1-\gamma^*$ . Hence, we conclude the proof.

Note that the matrix $\Sigma=U\Lambda U^{\mathsf{T}}$ where

\[\Lambda=\frac{n-m}{mn(n-1)}\begin{bmatrix}I_{n-1} &\quad 0_{n-1}\\0^{\mathsf{T}}_{n-1} &\quad 0\end{bmatrix},\]

and we can choose $U=[x_1,x_2,\ldots,x_n]$ , such that

(A1) \begin{align}x_i=\sqrt{\frac{n-i}{n-i+1}}\cdot \left(0,0,\ldots,1,-\frac{1}{n-i},-\frac{1}{n-i},\ldots,-\frac{1}{n-i}\right)^{\mathsf{T}},\end{align}

i.e. a scalar times first $i-1$ entries 0, 1 in the $i\textrm{th}$ entry and the rest $-1/(n-i)$ , for $1\le i\le n-1$ and $x_n=\tilde{1}/\sqrt{n}$ . In addition, note that

\[\Sigma^{1/2}=\sqrt{\frac{n-m}{mn(n-1)}}\cdot \left(I_n-\frac{1}{n}\tilde{1}\tilde{1}^{\mathsf{T}}\right).\]

Lemma 3. Let Assumption 5 hold. Then, we have

\[m^{3/2}\sum_{i=1}^{n}|w_i|^3 \xrightarrow[]{L^1} 0\]

as $n\to \infty$ with $\frac{m}{n} \to \gamma^*$ and $0\le \gamma^* \le 1$ .

Proof. We begin our proof by observing that $w_i$ all have the same distribution. This is easy to see, using the fact the $X_i$ are i.i.d. and

\[w_i=\sum_{j=1}^{n-1}\sqrt{\frac{n-m}{mn(n-1)}}\left(x_j^{\mathsf{T}}X\right)x_{j,i}+\frac{1}{n}\]

(this follows from $W=\Sigma^{1/2}X+1/n\,\tilde{1}$ ), where $x_j$ are as defined in (A1). In addition, note that we can take $X_i$ to have 0 mean as $\Sigma^{1/2}X=\Sigma^{1/2}\big(X-\mu \cdot \tilde{1}\big)+\Sigma^{1/2}\mu \cdot \tilde{1}=\Sigma^{1/2}\big(X-\mu \cdot \tilde{1}\big)$ . The last step follows as $\mu$ is a scalar and $\Sigma^{1/2}\tilde{1}=0$ . With this we have

\begin{align*} m^{3/2}\sum_{i=1}^{n}\left|w_i\right|^3 &= m^{3/2}\sum_{i=1}^{n}\left|\sum_{j=1}^{n-1}\sqrt{\frac{n-m}{mn(n-1)}}\left(x_j^{\mathsf{T}}X\right)x_{j,i}+\frac{1}{n}\right|^3\\ &\le 4 \left(\frac{n-m}{n(n-1)}\right)^{3/2}\sum_{i=1}^{n}\left|\sum_{j=1}^{n-1}\left(x_j^{\mathsf{T}}X\right)x_{j,i}\right|^3+ 4\frac{m^{3/2}}{n^2}. \end{align*}

It is easy to see that the second term, as for $0\le \gamma^* \le 1$ , converges to 0.

Define

\[T_i=\sum_{j=1}^{n-1}\left(x_j^{\mathsf{T}}X\right)x_{j,i}.\]

We need to check that

\[4 \left(\frac{n-m}{n(n-1)}\right)^{3/2}\sum_{i=1}^{n}\mathbb{E}|T_i|^3 \to 0 \ \textrm{as} \ n\to \infty.\]

We show $\frac{1}{n^{3/2}}\sum_{i=1}^{n}\mathbb{E}|T_i|^3 \to 0 \ \textrm{as} \ n\to \infty$ . First,

\[ \frac{1}{n^{3/2}}\sum_{i=1}^{n}\mathbb{E}|T_i|^3 = \frac{1}{\sqrt n} \,\mathbb{E}|T_1|^3,\]

which follows from the fact that $T_i$ are just centered and scaled $w_i$ and, hence, have the same distribution. In addition,

\begin{align*} T_i=\sum_{j=1}^{n-1}\left(x_j^{\mathsf{T}}X\right)x_{j,i} = \sum_{j=1}^{n-1}\left(\sum_{k=1}^{n}x_{j,k}X_k\right)x_{j,i} = \sum_{k=1}^{n}\left(\sum_{j=1}^{n-1}x_{j,i}x_{j,k}\right)X_k. \end{align*}

By using this and the Hoeffding inequality,

\[\mathbb{P}\left(|T_1|>t\right)\le 2\ \exp\left\{-\frac{ct^2}{K^2\sum_{k=1}^{n}\left(\sum_{j=1}^{n-1}x_{j,1}x_{j,k}\right)^2}\right\}\!,\]

where K is a positive constant that depends on the distribution of $X_1$ and $c>0$ is another such positive constant.

Note that our choice of $x_i$ ensures that $x_{j,i}=0$ when $j>i$ . Thus, in this case, $x_{1,1}$ is the only non-zero value. In addition, note that by our construction, $x_{1,1}=\sqrt{ (n-1)/n}$ . Thus,

\[\sum_{k=1}^{n}\left(\sum_{j=1}^{n-1}x_{j,1}x_{j,k}\right)^2=\left(1-\frac{1}{n}\right)\sum_{k=1}^{n}x_{j,k}^2=1-\frac{1}{n}.\]

This implies that

\[\mathbb{P}\left(|T_1|>t\right)\le 2\exp\left\{-\frac{ct^2}{K}\right\}\]

for some constants c, K. Hence, there exists some constant $C >0$ such that

\[\mathbb{E}|T_1|^3 \le C .\]

Hence, $\frac{1}{\sqrt n} \,\mathbb{E}|T_1|^3 \to 0 \ \textrm{as} \ n\to \infty$ . Hence, we conclude the proof for Assumption 5.

Lemma 4. Let Assumptions 3 and 5 hold. In the regime $\frac{m}{n}\to \gamma^* \ \textrm{with} \ 0\le \gamma^* \le 1$ , we have

\[\mathbb{E}\left[\frac{\sum_{i=1}^{n}|w_i|^3}{\left(\sum_{i=1}^{n}w^2_i\right)^{3/2}}\right]\to 0 \quad \textrm{as}\ n\to \infty .\]

Proof. We know the inequality

\[\sum_{i=1}^{n}a^p \le \left(\sum_{i=1}^{n}a_i\right)^p,\]

where $a_i \ge 0$ and $p>1$ . Using this, we know that

\[\frac{\sum_{i=1}^{n}|w_i|^3}{\left(\sum_{i=1}^{n}w^2_i\right)^{3/2}} \le 1 .\]

We also know from Lemmas 2 to 3, $m^{3/2}\sum_{i=1}^{n}|w_i|^3 \xrightarrow{P} 0 \ \textrm{as} \ n \to \infty$ and $m\sum_{i=1}^{n} w_i^2\xrightarrow{\textrm{a.s.}} 1 \ \textrm{as} \ n \to \infty$ . Hence,

\[\frac{\sum_{i=1}^{n}|w_i|^3}{\left(\sum_{i=1}^{n}w^2_i\right)^{3/2}} \xrightarrow{P} 0 \quad \textrm{as} \quad n \to \infty .\]

Using DCT, we are done.

Proof of Theorem 1. Let us consider $p=1$ , where p is the dimension. Consider,

\[\left|\mathbb{P}\left(\sqrt{m}\sum_{i=1}^{n}w_i(\nabla l(\theta,u_i)-\nabla g(\theta)) \le x\right)-\Phi_{\sigma}(x)\right|,\]

where $\Phi_{\sigma}(x)$ is the cumulative distribution function (CDF) of $N(0,\sigma^2(\theta))$ . Define $\left(\nabla l(\theta,u_i)-\nabla g(\theta)\right)=X_i$ . Thus, $X_i$ are i.i.d. with mean 0. For this particular case, we consider, without loss of generality, $\sigma^2(\theta)=\sigma^2=1$ . Therefore the problem reduces to proving

\[\left|\mathbb{P}\left(\sqrt{m}\sum_{i=1}^{n}w_i \ X_i \le x\right)-\Phi(x)\right|\]

goes to zero where $\Phi$ is the CDF of standard normal. Now,

\begin{align*} &\Big|\mathbb{E} \ \mathbb{P}\left(\sqrt{m}\sum_{i=1}^{n}w_i \ X_i \le x \mid w\right)-\mathbb{E} \left(\Phi(x)\right)\Big|\\ & \le \mathbb{E} \ \Big|\mathbb{P}\left(\sqrt{m}\sum_{i=1}^{n}w_i \ X_i \le x \mid w\right)- \Phi(x)\Big|\\ & \le \mathbb{E} \left[\ \frac{\sum_{i=1}^{n}\mathbb{E} \ [|w_i \ X_i|^3 \mid w]}{\left(\sum_{i=1}^{n}w_i^2\right)^{3/2}}\right]+\mathbb{E}\left|\Phi\left(\frac{x}{m\sum_{i=1}^{n}w^2_i}\right)-\Phi(x)\right|\\ & = \mathbb{E}|X_1|^3 \ \mathbb{E} \left[\frac{\sum_{i=1}^{n}|w_i| ^3}{\left(\sum_{i=1}^{n}w_i^2\right)^{3/2}}\right] +\mathbb{E}\left|\Phi\left(\frac{x}{m\sum_{i=1}^{n}w^2_i}\right)-\Phi(x)\right|. \end{align*}

Now, it has been proved in Lemma 4 that the first term goes to 0. Using $m\sum_{i=1}^{n}w^2_i \xrightarrow{\textrm{a.s.}} 1 \ \textrm{as} \ n \to \infty$ and the fact that $||\Phi||_{\infty}<1$ , we get that the second term converges to zero as well using DCT. We can extend to any dimension using the Cramer–Wold device. Hence, the proof is complete.

Appendix B. Proofs for positive weights with Dirichlet example

Theorem 7. [Reference Arenal-Gutiérrez and Matrán1, Corollary 3.1]. Let $X_n; \ n=1,2,\ldots$ be a sequence of random variables, which are i.i.d. with $Var(X_n)=\sigma^2$ . If $w_n$ is a sequence of weight vectors satisfying Assumption 6, then

\[\sqrt{m}\left(\sum_{i=1}^{n}w_{n,i} X_i -\frac{1}{n}\sum_{i=1}^{n}X_i\right)\overset{d}{\rightarrow} N(0,c^2\,\sigma^2).\]

Proof of Theorem 2. The proof follows easily from Theorem 7 by noting that

\begin{align*} \sqrt{m}\left(\sum_{i=1}^{n}w_{n,i} X_i -\frac{1}{n}\sum_{i=1}^{n}X_i\right)=\sqrt{m}\sum_{i=1}^{n}w_{n,i} X_i-\sqrt{\frac{m}{n}}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}X_i. \end{align*}

We know that since $X_i$ are i.i.d. mean zero with second moment and $m/n \to 0$ , $\sqrt{\frac{m}{n}}\frac{1}{\sqrt{n}}\sum_{i=1}^{n}X_i \overset{P}{\rightarrow} 0$ as $n\to \infty$ . Therefore, the result follows using Slutsky’s theorem.

Proposition 4. If the random variables $w_i$ are exchangeable and $w_i \ge 0$ with

\[\mathbb{E}(w^3_i) \le \frac{o\left(m^{-3/2}\right)}{n}, \quad \mathbb{E}\big(w^4_i\big)\le \frac{o(m^{-2})}{n} \quad \textrm{and} \quad \mathbb{E}\left(w^2_i w^2_j\right)=\left(\frac{1}{m\,n}\right)^2+o\left(\left(\frac{1}{m\,n}\right)^2\right)\!,\]

then Assumption 6 holds if $m/n \to 0$ as $n\to \infty$ .

Proof of Proposition 4. Note that we only need to establish the last two points in Assumption 6. For the first point we have for any $\epsilon$

\begin{align*} \mathbb{P}\left(\sqrt{m}\max_{1\le \, j\le n}\left|w_i -\frac{1}{n}\right|\ge \epsilon\right)&\le n\, \mathbb{P}\left(\sqrt{m}\left|w_1-\frac{1}{n}\right|\ge \epsilon\right)\\ &\le \frac{1}{\epsilon^3}\,n\, m^{3/2}\, \mathbb{E}\left|w_i -\frac{1}{n}\right|^3\\ &\le \frac{4\,n\, m^{3/2}}{\epsilon^3} \left(\mathbb{E}\left|w_i\right|^3 +\frac{1}{n^3}\right). \end{align*}

By our hypothesis the right-hand side goes to 0 as $n\to \infty$ .

For the second condition, define $Y_i=m \ w_i$ . Using this and the fact that $w_i$ are exchangeable, we have

\begin{align*} \mathbb{E}\left(\frac{1}{m}\sum_{i=1}^{n}Y^2_i\right)=\frac{n}{m} \mathbb{E}(Y^2_1)=1 \end{align*}

and

\begin{align*} \textrm{Var}\left(\frac{1}{m}\sum_{i=1}^{n}Y^2\right)&=\mathbb{E}\left(\frac{1}{m}\sum_{i=1}^{n}Y^2_i\right)^2-\left[\mathbb{E}\left(\frac{1}{m}\sum_{i=1}^{n}Y^2_i\right)\right]^2\\ & = \mathbb{E}\left(\frac{1}{m}\sum_{i=1}^{n}Y^2_i\right)^2-1 \\ &=\frac{1}{m^2} \,\mathbb{E}\left(\sum_{i=1}^{n}Y^4_i\right)+\frac{1}{m^2} \, \mathbb{E}\left(2\sum_{1\le i<j \le n}Y^2_i Y^2_j\right)-1\\ &=\frac{n}{m^2} \,\mathbb{E}(Y^4_1)+\frac{n(n-1)}{m^2} \,\mathbb{E}(Y^2_1 Y^2_2)-1\\ &\le \frac{o(m^2)}{m^2}+\frac{n(n-1)}{m^2}\left(o\left(\left(\frac{m}{n}\right)^2\right)+\left(\frac{m}{n}\right)^2\right)-1. \end{align*}

Therefore, $m\sum_{i=1}^{n}w^2_i \overset{P}{\rightarrow} 1$ as $n\to \infty$ . This implies

\begin{align*} m\sum_{i=1}^{n} \left(w_i-\frac{1}{n}\right)^2&=m\, \sum_{i=1}^{n}w^2_i-\frac{2\,m}{n}\sum_{i=1}^{n}w_i+\frac{m}{n}\\ &=m\, \sum_{i=1}^{n}w^2_i-\frac{m}{n}. \end{align*}

Using the fact that $m/n \to 0$ as $n\to \infty$ and $m\sum_{i=1}^{n}w^2_i \overset{P}{\rightarrow} 1$ , we are done.

Proof of Proposition 1. We shall make use of Proposition 4 and the following lemma, which we state without proof as the proof of the lemma is simple.

Lemma 5. If $X\sim Dir(\alpha)$ , where $\alpha=(\alpha_1,\alpha_2,\ldots,\alpha_n)$ , then

\[\mathbb{E}\left(\prod_{i=1}^{n}X^{\beta_i}_i\right)=\frac{\Gamma\left(\sum_{i=1}^{n}\alpha_i\right)}{\Gamma\left(\sum_{i=1}^{n}(\alpha_i+\beta_i)\right)}\times \prod_{i=1}^{n}\frac{\Gamma\left(\alpha_i+\beta_i\right)}{\Gamma(\alpha_i)}.\]

We start with the third moment

\begin{align*} \mathbb{E}(Y^3_i)&=m^3 \,\mathbb{E}(w^3_i)=m^3 \,\mathbb{E}(w^3_1)\\ &=m^3\frac{\Gamma\left(\sum_{i=1}^{n}\frac{m-1}{n-m}\right)}{\Gamma\left(\sum_{i=1}^{n}\frac{m-1}{n-m}+3\right)}\cdot \frac{\Gamma\left(\frac{m-1}{n-m}+3\right)}{\Gamma\left(\frac{m-1}{n-m}\right)}\\ & =\frac{m^3}{n}\frac{\left(\frac{m-1}{n-m}+2\right)\left(\frac{m-1}{n-m}+1\right)}{\left(\frac{n(m-1)}{n-m}+2\right)\left(\frac{n(m-1)}{n-m}+1\right)}. \end{align*}

This implies

\begin{align*} \mathbb{E}(Y^3_i) & \le \frac{m^3}{n} \left(\frac{1}{n}+2\frac{n-m}{n(m-1)}\right)\left(\frac{1}{n}+\frac{n-m}{n(m-1)}\right)\\ & \le \frac{m}{n} \left(\frac{m}{n}+2\frac{m(n-m)}{n(m-1)}\right)\left(\frac{m}{n}+\frac{m(n-m)}{n(m-1)}\right)\\ &=O\left(\frac{m}{n}\right)\!, \end{align*}

where the second step follows from Lemma 5. Hence we establish $\mathbb{E}(Y^3_i) \le o(m^{3/2})/n$ . We can similarly argue for $\mathbb{E}(Y^4_i)$ . In fact,

\begin{align*} \mathbb{E}(Y^4_i)&=m^4 \,\mathbb{E}(w^4_i)=m^4 \,\mathbb{E}(w^4_1)\\ &=m^4\frac{\Gamma\left(\sum_{i=1}^{n}\frac{m-1}{n-m}\right)}{\Gamma\left(\sum_{i=1}^{n}\frac{m-1}{n-m}+4\right)}\cdot \frac{\Gamma\left(\frac{m-1}{n-m}+4\right)}{\Gamma\left(\frac{m-1}{n-m}\right)}. \end{align*}

This implies that

\begin{align*} \mathbb{E}(Y^4_i) & =\frac{m^4}{n}\frac{\left(\frac{m-1}{n-m}+3\right)\left(\frac{m-1}{n-m}+2\right)\left(\frac{m-1}{n-m}+1\right)}{\left(\frac{n(m-1)}{n-m}+3\right)\left(\frac{n(m-1)}{n-m}+2\right)\left(\frac{n(m-1)}{n-m}+1\right)}\\ & \le \frac{m^4}{n} \left(\frac{1}{n}+3\frac{n-m}{n(m-1)}\right) \left(\frac{1}{n}+2\frac{n-m}{n(m-1)}\right)\left(\frac{1}{n}+\frac{n-m}{n(m-1)}\right)\\ & =\frac{m}{n} \left(\frac{m}{n}+3\frac{m(n-m)}{n(m-1)}\right)\left(\frac{m}{n}+2\frac{m(n-m)}{n(m-1)}\right)\left(\frac{m}{n}+\frac{m(n-m)}{n(m-1)}\right)\\ &=O\left(\frac{m}{n}\right). \end{align*}

Hence, we establish the required condition for the fourth power as $\mathbb{E}(Y^4_i)=o(m^2)/n$ . The last step is to show that $\mathbb{E}(Y^2_iY^2_j)$ satisfies the assumption. Now

\begin{align*} \mathbb{E}(Y^2_iY^2_j)&=m^4 \,\mathbb{E}(w^2_iw^2_j)=m^4 \,\mathbb{E}(w^2_1w^2_2)\\ &=m^4\frac{\Gamma\left(\sum_{i=1}^{n}\frac{m-1}{n-m}\right)}{\Gamma\left(\sum_{i=1}^{n}\frac{m-1}{n-m}+4\right)}\cdot \left[\frac{\Gamma\left(\frac{m-1}{n-m}+2\right)}{\Gamma\left(\frac{m-1}{n-m}\right)}\right]^2. \end{align*}

This leads to

\begin{align*} \mathbb{E}(Y^2_iY^2_j) & =m^4\frac{\left(\frac{m-1}{n-m}+1\right)^2\left(\frac{m-1}{n-m}\right)^2}{\left(\frac{n(m-1)}{n-m}+3\right)\left(\frac{n(m-1)}{n-m}+2\right)\left(\frac{n(m-1)}{n-m}+1\right)\frac{n(m-1)}{n-m}}\\ & \le \frac{m^4}{n^2} \left(\frac{1}{n}+\frac{n-m}{n(m-1)}\right)^2\\ & =\frac{m^2}{n^2}\left[\left(\frac{m}{n}+\frac{m(n-m)}{n(m-1)}\right)^2-1\right]+\frac{m^2}{n^2} \\ &=o\left(\frac{m^2}{n^2}\right)+\frac{m^2}{n^2}. \end{align*}

Hence, Assumption 6 is satisfied. This implies that the CLT holds for Dirichlet weights with the parameter vector $(\frac{m-1}{n-m},\frac{m-1}{n-m},\ldots,\frac{m-1}{n-m})^{\mathsf{T}}$ .

Appendix C. Proofs for the general regime

Now we take a close look at the M-SGD algorithm. Consider the gradient descent and its continuous version given as

(C1) \begin{align}\tilde x_{k+1}&=\tilde x_k-\gamma \nabla g(\tilde x_k),\end{align}

(C2) \begin{align}\textrm{d}\tilde{X}_t&=-\nabla g(\tilde{X}_t)\,\textrm{d}t.\\[6pt] \nonumber\end{align}

We start by showing a result about (C1) and (C2).

Lemma 6. Under Assumption 1, for the gradient descent algorithm $\tilde x_k$ defined by (C1) and its continuous version $\tilde{X}_t$ defined by (C2), we have

\begin{align*} \left|\tilde{x}_k-\tilde{X}_{k\gamma}\right|\le C_1k\gamma \, \left(1+L\gamma\right)^{k}. \end{align*}

Proof. Note that $\tilde{X_0}=\tilde{x_0}$ by our implicit assumption that all starting points are the same. Let $t \in [0,\gamma]$ . Note that

\begin{align*} \tilde X_t&=\tilde X_0-\int_{0}^{t}\nabla g(\tilde X_s) \,\textrm{d}s;\\ \tilde x_1&=\tilde{x_0}-\gamma\nabla g(\tilde{x_0}). \end{align*}

Hence, we have

\begin{align*} \left|\tilde{X_t}-\tilde{x_1}\right|&\le \int_{0}^{t}\left|\nabla g(\tilde{X_s})-\nabla g(\tilde{x_0})\right|+\left(\gamma-t\right)\left|\nabla g(\tilde x_0)\right|\\ &\le L\int_{0}^{t}\left|\tilde X_s-\tilde x_0\right|+\gamma \left|\nabla g(\tilde{x_0})\right|. \end{align*}

Therefore,

\begin{align*} \left|\tilde{X_t}-\tilde{x_1}\right|& \le L\int_{0}^{t}\left|\tilde X_s-\tilde x_1\right|+Lt\left|\tilde{x_1}-\tilde{x_0}\right|+\gamma \left|\nabla g(\tilde{x_0})\right|\\ &\le L\int_{0}^{t}\left|\tilde X_s-\tilde x_1\right|+L\gamma^2\left|\nabla g(\tilde{x_0})\right|+\gamma \left|\nabla g(\tilde{x_0})\right|\\ &\le \left|\nabla g(\tilde{x_0})\right|\gamma \left(1+L\gamma\right)+ L\int_{0}^{t}\left|\tilde X_s-\tilde x_1\right|. \end{align*}

Using the Gronwall lemma, we get

\begin{align*} \left|\tilde{X_t}-\tilde{x_1}\right|&\le \left|\nabla g(\tilde{x_0})\right|\gamma \left(1+L\gamma\right) e^{L\gamma}. \end{align*}

Note that as we are in the regime $0\le k\le [T/\gamma]$ , we can bound $e^{L\gamma}$ by $e^T$ where as stated before T is fixed. Let the induction step in $t \in [(k-1)\gamma,k\gamma]$ hold as

\begin{align*} \left|\tilde X_t-\tilde x_k\right|\le \left|\nabla g(\tilde{x_0})\right|k\gamma e^{kL\gamma}\left(1+\gamma L\right)^{k}. \end{align*}

Let $t \in [k\gamma,(k+1)\gamma]$ . Note that

\begin{align*} \left|\tilde X_t-\tilde x_{k+1}\right|& \le \left|\tilde X_{k\gamma}-\tilde x_k\right|+L\int_{k\gamma}^{t}\left|\tilde X_s-\tilde x_k\right| \,\textrm{d}s+\left((k+1)\gamma-t\right)\left|\nabla g(\tilde x_k)\right|\\ &\le \left|\nabla g(\tilde{x_0})\right|k\gamma e^{kL\gamma}\left(1+\gamma L\right)^{k}+L\int_{k\gamma}^{t}\left|\tilde X_s-\tilde x_{k+1}\right|+\left(1+L\gamma\right)\gamma\left|\nabla g(\tilde x_k)\right|. \end{align*}

In addition, noting that $\left|\nabla g(\tilde{x}_k)\right|\le \left(1+\gamma L\right)^k \left|\nabla g(\tilde{x}_0)\right|$ as

\[\left|\nabla g(\tilde {x}_{k+1})\right|\le L\,\left|\tilde{x}_{k+1}-\tilde{x}_k\right|+\left|\nabla g(\tilde{x}_k)\right|\le L\gamma \left|\nabla g(\tilde{x}_k)\right| +\left|\nabla g(\tilde{x}_k)\right|=\left(1+L\gamma\right)\left|\nabla g(\tilde{x}_k)\right|\]

and rearranging some terms we can see that

\begin{align*} \left|\tilde X_t-\tilde x_{k+1}\right|\le \left|\nabla g(\tilde{x_0})\right|\left(k+1\right)\gamma e^{kL\gamma}\left(1+\gamma L\right)^{k+1} +L\int_{k\gamma}^{t}\left|\tilde X_s-\tilde x_{k+1}\right|. \end{align*}

Using the Gronwall inequality the induction step is completed. This completes the proof.

Lemma 7. For any $a,b\ge 0, \,x\ge 1$ , we have

\begin{align*} \left(1+\frac{a}{x}+\frac{b}{x^2}\right)^x \le \exp\left(a+b+1\right). \end{align*}

The proof of Lemma 7 is trivial; therefore, we skip the proof.

Lemma 8. Under Assumptions 1–4, we have for the algorithms described in (4) and (C1),

\begin{align*} \mathbb{E}\left|x_{k+1}-\tilde{x}_{k+1}\right|^2 \le \tilde{C}^*_1 \frac{\gamma^2}{m}+\tilde{C}^*_2 \frac{k}{m}, \end{align*}

where

\begin{align*} \tilde{C}^*_1&= e^{\pi^2/6}\exp\left(1+2TL+T^2L^2+2TL_1p+2T^2LL_1p+T^2p^2L^2_1\right)\left|\left|\sigma(x_0)\right|\right|^2_F\\ &\quad \textrm{and}\\ \tilde{C}^*_2&=\frac{\tilde{C}^*_1}{\left\|\sigma(x_0)\right\|_F^2}\big(T^2+1\big) \big(\ 2\, L^2_1\, p\, C^2_1 T^2 e^{2LT} + 2\sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\big). \end{align*}

Proof of Lemma 8. Now that

\begin{align*} x_{k+1}-\tilde{x}_{k+1}&=\left(x_{k}-\tilde{x}_{k}\right)-\gamma\left(\nabla g(x_k)-\nabla g(\tilde{x}_k)\right)+\sqrt{\frac{\gamma}{m}}\sigma(x_k)\sqrt\gamma \xi_{k+1}. \end{align*}

This implies that

\begin{align*} \left|x_{k+1}-\tilde{x}_{k+1}\right|&\le \left|x_{k}-\tilde{x}_{k}\right|+\gamma\left|\nabla g(x_k)-\nabla g(\tilde{x}_k) \right|+\sqrt{\frac{\gamma}{m}}\left|\sigma(x_k)\sqrt\gamma \xi_{k+1}\right|. \end{align*}

Using the fact that $\nabla g$ is Lipschitz, we have

\begin{align*} \left|x_{k+1}-\tilde{x}_{k+1}\right|&\le \left|x_{k}-\tilde{x}_{k}\right|+\gamma L\left|x_k-\tilde{x}_k \right|+\frac{\gamma}{\sqrt m}\left|\sigma(x_k)\xi_{k+1}\right|;\\ \left|x_{k+1}-\tilde{x}_{k+1}\right|&\le \left(1+\gamma L\right)\left|x_k-\tilde{x}_k \right|+\frac{\gamma}{\sqrt m}\left|\left(\sigma(x_k)-\sigma(\tilde{x}_k)\right)\xi_{k+1}\right|+\frac{\gamma}{\sqrt m}\left|\sigma(\tilde{x}_k)\xi_{k+1}\right|. \end{align*}

In addition, using the fact $\left|\left(\sigma(x_k)-\sigma(\tilde{x}_k)\right)\xi_{k+1}\right|\le \left|\sigma(x_k)-\sigma(\tilde{x}_k)\right|_2\left|\xi_{k+1}\right|$ and

Assumption 2 above, we have

\begin{align*} \left|x_{k+1}-\tilde{x}_{k+1}\right|&\le \left(1+\gamma L\right)\left|x_k-\tilde{x}_k \right|+\frac{\gamma}{\sqrt m}\left|\left(\sigma(x_k)-\sigma(\tilde{x}_k)\right)\xi_{k+1}\right|+\frac{\gamma}{\sqrt m}\left|\sigma(\tilde{x}_k)\xi_{k+1}\right|;\\ \left|x_{k+1}-\tilde{x}_{k+1}\right|&\le \left(1+\gamma L\right)\left|x_k-\tilde{x}_k \right|+\frac{\gamma}{\sqrt m}L_1 \left|x_k-\tilde{x}_k\right|\left|\xi_{k+1}\right|+\frac{\gamma}{\sqrt m}\left|\sigma(\tilde{x}_k)\xi_{k+1}\right|\\ &\le \left[\left(1+\gamma L\right) + \frac{\gamma}{\sqrt{m}}L_1\left|\xi_{k+1}\right|\right]\left|x_k-\tilde{x}_k\right|+\frac{\gamma}{\sqrt{m}}\left|\sigma(\tilde{x}_k)\xi_{k+1}\right|. \end{align*}

Square the above and then use Jensen’s inequality to see

\begin{align*} &\left|x_{k+1}-\tilde{x}_{k+1}\right|^2\le \left(1+\frac{1}{k^2}\right)\left[\left(1+\gamma L\right) + \frac{\gamma}{\sqrt{m}}L_1\left|\xi_{k+1}\right|\right]^2 \left|x_k-\tilde{x}_k \right|^2\\ &\quad \quad \quad \quad \quad \quad\quad \quad \quad \quad +\frac{\left(1+k^2\right)\gamma^2}{m}\left|\sigma(\tilde{x}_k)\xi_{k+1}\right|^2. \end{align*}

By taking expectation, we get

\begin{align*} &\mathbb{E}\left|x_{k+1}-\tilde{x}_{k+1}\right|^2\\ &\le \left(1+\frac{1}{k^2}\right)\mathbb{E}\left[\left(1+\gamma L\right) + \frac{\gamma}{\sqrt{m}}L_1\left|\xi_{k+1}\right|\right]^2 \mathbb{E}\left|x_k-\tilde{x}_k \right|^2+\frac{\left(1+k^2\right)\gamma^2}{m}\left|\sigma(\tilde{x}_k)\xi_{k+1}\right|^2\\ & \le \left[\left(1+\gamma L\right)^2+\frac{2\gamma\,\sqrt{p}}{\sqrt{m}}L_1 \left(1+\gamma L\right)+\frac{\gamma^2}{m}p\,L^2_1\right]^k\Bigg\{\left[\prod_{j=1}^{k}\left(1+\frac{1}{j^2}\right)\right]\mathbb{E}\left|x_1-\tilde{x}_1\right|^2\\ &\quad +\sum_{j=1}^{k-1}\prod_{i=1}^{j}\left(1+\frac{1}{\left(k-i+1\right)^2}\right)\frac{\gamma^2\left((k-j)^2+1\right)}{m}\left|\left|\sigma(\tilde{x}_{k-j})\right|\right|^2_F\Bigg\}\\ &\quad +\frac{\gamma^2\left(k^2+1\right)}{m}\left|\left|\sigma(\tilde x_k)\right|\right|^2_F. \end{align*}

Using the fact that the arithmetic mean of n positive real numbers is always greater than the geometric mean of the same real numbers, for any $k \in \mathbb{N}$ , we obtain

\begin{align*} \prod_{j=1}^{k}\left(1+\frac{1}{j^2}\right) \le \left(1+\frac{1}{k}\sum_{j=1}^{k}\frac{1}{j^2}\right)^k. \end{align*}

In addition, $\sum_{j=1}^{k}\frac{1}{j^2} \le \sum_{j=1}^{\infty} \frac{1}{j^2}=\frac{\pi^2}{6}$ . Therefore, we have

\begin{align*} \prod_{j=1}^{k}\left(1+\frac{1}{j^2}\right) &\le \left(1+\frac{\pi^2}{6k}\right)^k\\ &\le e^{\pi^2/6}. \end{align*}

In addition, note that

\begin{align*} \mathbb{E}\left|x_1-\tilde{x}_1\right|^2=\frac{\gamma^2}{m}\left|\left|\sigma(x_0)\right|\right|^2_F. \end{align*}

Therefore, using Lemma 7 the first term is less than

\begin{align*} e^{\pi^2/6}\exp\left(1+2TL+T^2L^2+2TL_1\,\sqrt{p}+2T^2LL_1\,\sqrt{p}+T^2p\,L^2_1\right)\frac{\gamma^2}{m} \,\mathbb{E}\left|\left|\sigma(x_0)\right|\right|^2_F=\tilde{C}^*_1 \frac{\gamma^2}{m}. \end{align*}

Let us define

\[\tilde{L}^*=\exp\left(1+2TL+T^2L^2+2TL_1\,\sqrt{p}+2T^2LL_1\,\sqrt{p}+T^2p\,L^2_1\right).\]

It can be seen that the second term is bounded by

\begin{align*} & \tilde{L}^*\,\frac{T^2+1}{m}\sup\limits_{0\le k\le T/\gamma} \left|\left|\sigma(\tilde x_k)\right|\right|^2_F \cdot k\prod_{j=1}^{k}\left(1+\frac{1}{j^2}\right)\\ &\le e^{\pi^2/6}\frac{\left(T^2+1\right)k}{m}\sup\limits_{0\le k\le T/\gamma} \left|\left|\sigma(\tilde x_k)\right|\right|^2_F. \end{align*}

Observe

\begin{align*} \left(\textrm{Tr} \sigma^2(\tilde{x}_{k})\right)^{1/2} &=\left|\left|\sigma(\tilde{x}_{k})\right|\right|_F\\ & \le \left|\left|\sigma(\tilde{x}_{k})-\sigma(\tilde{X}_{k\gamma})\right|\right|_F+\left|\left|\sigma(\tilde{X}_{k\gamma})\right|\right|_F\\ &\le L_1 \sqrt{p}\left|\tilde{x}_k-\tilde{X}_{k\gamma}\right|+\sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|_F\\ & \le L_1 \sqrt{p} \ C_1 k \gamma \left(1+L\gamma\right)^{k} + \sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|_F. \end{align*}

The second inequality follows from the relation between spectral norm and Frobenius norm and the last one follows from the fact that the discretized version of gradient descent is only an order of the step size away from its continuous counterpart. Thus,

\begin{align*} \sup\limits_{0\le k\le T/\gamma} \left|\left|\sigma(\tilde x_k)\right|\right|^2_F \le 2 L^2_1 p \ C^2_1 T^2 e^{2\,LT} + 2\sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F. \end{align*}

Therefore, the second term is less than

\begin{align*} \tilde{L}^*\,e^{\pi^2/6}\frac{k\left(T^2+1\right)}{m} \left(2 \, L^2_1\,p\,C^2_1 T^2 e^{2\,LT} + 2\sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\right)=\tilde{C}^*_{2} \frac{k}{m}. \end{align*}

As a result

\begin{align*} E\left|x_{k+1}-\tilde{x}_{k+1}\right|^2 \le \tilde{C}^*_1 \frac{\gamma^2}{m}+\tilde{C}^*_2 \frac{k}{m}. \end{align*}

Note: Using Lemma 7, we have the rate in the previous lemma as $\frac{k}{m}$ . This is because all the other terms are bounded in the regime $\gamma K=T$ .

Next we show (5) and (C1) are close in the Wasserstein distance. To show the above statement we need one more lemma.

Lemma 9. Under Assumptions 1–4, we have

\begin{align*} \mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2 \le 2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\mathbb{E}\left|x_{k,n}-\tilde{x}_k\right|^2. \end{align*}

Proof. We start with bounding the second moment:

\begin{align*} &\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2 \\& \le \mathbb{E}\Bigg(\left|\sum_{i=1}^{n}\left(w_{i,k}-\frac{1}{n}\right)(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k}))\right|\\ &\quad +\left|\sum_{i=1}^{n}\frac{1}{n}(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k}))\right|\Bigg)^2. \end{align*}

Thus,

\begin{align*} &\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2 \\ &\le 2 \, \mathbb{E}\left|\sum_{i=1}^{n}\left(w_{i,k}-\frac{1}{n}\right)(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k}))\right|^2\\ &\quad + 2 \, \mathbb{E}\left(\frac{1}{n}\sum_{i=1}^{n}h_1(u_i)\left|x_{n,k}-\tilde{x}_k\right|\right)^2. \end{align*}

Hence,

\begin{align*} &\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2 \\ &\le 2 \, \mathbb{E}\Bigg(\sum_{i=1}^{n}\left(w_{i,k}-\frac{1}{n}\right)^2\left|\nabla l(x_{n,k},u_{i,k})-\nabla l(\tilde{x}_k,u_{i,k})\right|^2\\ &\quad +\sum_{i\ne j}\left(w_{i,k}-\frac{1}{n}\right)\left(w_{j,k}-\frac{1}{n}\right)\\ &\quad\quad\left(\nabla l(x_{n,k},u_{i,k})-\nabla l(\tilde{x}_k,u_{i,k})\right)^{\mathsf{T}}\left(\nabla l(x_{n,k},u_{j,k})-\nabla l(\tilde{x}_k,u_{j,k})\right)\Bigg)\\ &\quad+2 \, \mathbb{E}\left(\frac{1}{n}\sum_{i=1}^{n}h_1(u_{i,k})\right)^2 \mathbb{E}\left|x_{n,k}-\tilde{x}_k\right|^2. \end{align*}

Therefore,

\begin{align*} &\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2\\&\le 2 \Bigg(\sum_{i=1}^{n}\frac{n-m}{mn^2} \,\mathbb{E}\left|\nabla l(x_{n,k},u_{i,k})-\nabla l(\tilde{x}_k,u_{i,k})\right|^2\\ &\quad - \sum_{i\ne j} \frac{n-m}{mn^2(n-1)} \,\mathbb{E}\left|\nabla g(x_{n,k})-\nabla g(\tilde{x}_k)\right|^2\Bigg)\\ &\quad + 2\,\mathbb{E}(h_1(u))^2\,\mathbb{E}\left|x_{n,k}-\tilde{x}_k\right|^2 \end{align*}

using the definition of the weights $w_i$ and Jensen’s inequality. This concludes

\begin{align*} &\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2 \\&\le 2 \Bigg(\sum_{i=1}^{n}\frac{n-m}{mn^2} \,\mathbb{E}\left|\nabla l(x_{n,k},u_{i,k})-\nabla l(\tilde{x}_k,u_{i,k})\right|^2\Bigg)\\ &\quad +2\, \,\mathbb{E}(h_1(u))^2\mathbb{E}\left|x_{n,k}-\tilde{x}_k\right|^2. \end{align*}

This implies

\begin{align*} & \mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})- \nabla l(\tilde{x}_k,u_{i,k})\right)\right|^2 \\&\le 2 \Bigg(\sum_{i=1}^{n}\frac{n-m}{mn^2}\mathbb{E}\left(h_1^2(u)\right)\mathbb{E}\left|x_{n,k}-\tilde{x}_k\right|^2\Bigg)\\ &\quad +2\, \mathbb{E}(h_1(u))^2\,\mathbb{E}\left|x_{k,n}-\tilde{x}_k\right|^2\\ & \le 2\left(\frac{n-m}{mn}+1\right) \mathbb{E}\left(h_1^2(u)\right) \mathbb{E}\left|x_{n,k}-\tilde{x}_k\right|^2 .\end{align*}

Lemma 10. With $x_{n,k}$ and $\tilde x_k$ as defined in (5) and (C1), under Assumptions 1–4,

\begin{align*} \mathbb{E}\left|x_{n,k+1}-\tilde x_{k+1}\right|^2 &\le K^*_1 \,\frac{3^k\gamma}{m}, \end{align*}

where

\begin{align*} K^{*}_1=T\cdot \exp\left(1+6T^2 \mathbb{E}\left(h_1^2(u)\right)\right) \cdot \left(2p^2L^2_1 C^2_1 T^2 e^{2TL}+2\sup\limits_{0\le t\le T}||\sigma(\tilde{X}_t)||^2_F\right). \end{align*}

Proof. Recall the definitions of (5) and (C1). Then

\begin{align*} x_{n,k+1}-\tilde x_{k+1}&=x_{n,k}-\tilde x_{k}-\gamma \sum_{i=1}^{n}w_{i,k}\left(\nabla l\left(x_{n,k},u_{i,k}\right)-g(\tilde x_k)\right)\\ &=x_{n,k}-\tilde x_{k}-\gamma\sum_{i=1}^{n}w_{i,k}\left(\nabla l\left(x_{n,k},u_{i,k}\right)-\nabla l(\tilde{x}_k,u_{i,k}\right)\\&\quad -\gamma\sum_{i=1}^{n}\left(\nabla l(\tilde{x}_k,u_{i,k}-\gamma g(\tilde x_k)\right). \end{align*}

This implies that

\begin{align*} &\left| x_{n,k+1}-\tilde x_{k+1}\right|\\ & \le \left|x_{n,k}-\tilde x_{k}\right| +\gamma \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_i)-\nabla l(\tilde{x}_k,u_i)\right)\right|+\gamma \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(\tilde{x}_k,u_i)-\nabla g(\tilde{x}_k)\right)\right|. \end{align*}

Square both sides of the above and use Jensen’s inequality to get

\begin{align*} \left| x_{n,k+1}-\tilde x_{k+1}\right|^2&\le 3 \left|x_{n,k}-\tilde x_{k}\right|^2 +3\gamma^2 \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_i)-\nabla l(\tilde{x}_k,u_i)\right)\right|^2\\&\quad +3\gamma^2 \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(\tilde{x}_k,u_i)-\nabla g(\tilde{x}_k)\right)\right|^2. \end{align*}

Thus, taking expectation, we have

\begin{align*} \mathbb{E}\left|x_{n,k+1}-\tilde x_{k+1}\right|^2 &\le 3 \, \mathbb{E}\left|x_{n,k}-\tilde x_{k}\right|^2+3\gamma^2 \mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_i)-\nabla l(\tilde{x}_k,u_i)\right)\right|^2\\&\quad +3\gamma^2 \mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(\tilde{x}_k,u_i)-\nabla g(\tilde{x}_k)\right)\right|^2. \end{align*}

This implies that

\begin{align*} &\mathbb{E}\left|x_{n,k+1}-\tilde x_{k+1}\right|^2\\ &\le 3 \,\mathbb{E}\left|x_{n,k}-\tilde x_{k}\right|^2 + 3\gamma^2 2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\mathbb{E}\left|x_{n,k}-\tilde x_{k}\right|^2+\frac{3\gamma^2}{m}||\sigma(\tilde{x}_k)||^2_F\\ &=3\left[1+2\gamma^2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\right]\mathbb{E}\left|x_{n,k}-\tilde x_{k}\right|^2+\frac{3\gamma^2}{m}||\sigma(\tilde{x}_k)||^2_F. \end{align*}

Here the last inequality follows from the previous lemma. Continuing from the last line, we get

\begin{align*} &\mathbb{E}\left| x_{n,k+1}-\tilde x_{k+1}\right|^2 \\&\le \sum_{j=0}^{k} 3^{j+1}\left[1+ 2\gamma^2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\right]^{j} \frac{\gamma^2}{m}||\sigma(\tilde{x}_{k-j})||^2_F\\ &\le\frac{\gamma^2}{m}\ \left(k+1\right) \ 3^{k+1} \left[1 + 2\gamma^2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\right]^{k} \sup\limits_{0\le k \le [\frac{T}{\gamma}]} ||\sigma(\tilde{x}_k)||^2_F\\ & \le \frac{\gamma^2}{m}\ \left(k+1\right) \ 3^{k+1} \left[1 + 2\gamma^2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\right]^{k}\\ &\quad \left[2p^2L^2_1C^2_1k^2\gamma^2\left(1+L\gamma\right)^{2k}+2\sup\limits_{0\le t\le T}||\sigma(\tilde{X}_t)||^2_F\right]. \end{align*}

From Lemma 7, we see that

\begin{align*} \left[1+ 2\gamma^2\left(\frac{n-m}{mn}+1\right)\mathbb{E}\left(h_1^2(u)\right)\right]^{k} \le \exp\left(1+6T^2 \,\mathbb{E}\left(h_1^2(u)\right)\right). \end{align*}

We use the fact $\left(\frac{n-m}{mn}+1\right)\le 3$ in the above bound. In addition, using the fact $K\gamma=T$ , we have

\begin{align*} \left[2p^2L^2_1C^2_1k^2\gamma^2\left(1+L\gamma\right)^{2k}+2\sup\limits_{0\le t\le T}||\sigma(\tilde{X}_t)||^2_F\right]\le 2p^2L^2_1 C^2_1 T^2 e^{2TL}+2\sup\limits_{0\le t\le T}||\sigma(\tilde{X}_t)||^2_F. \end{align*}

Hence,

\begin{align*} \mathbb{E}\left| x_{n,k+1}-\tilde x_{k+1}\right|^2 \le K^{*}_1 \frac{3^k \gamma}{m}, \end{align*}

where

\begin{align*} K^{*}_1=T\cdot \exp\left(1+6T^2 \mathbb{E}(h_1^2(u))\right) \cdot \left(2p^2L^2_1 C^2_1 T^2 e^{2TL}+2\sup\limits_{0\le t\le T}||\sigma(\tilde{X}_t)||^2_F\right). \end{align*}

Thus, we conclude the proof.

Proposition 5. Suppose Assumptions 1–4 hold. Recall $x_{n,k}$ and $x_k$ from (5) and (4). Then for any $k \le K= \left[T/\gamma\right]$ with $m\ge 3^{K}$ ,

\begin{align*} \mathbb{E}\left|x_{n,k}-x_{k}\right|^2 &\le K_1\cdot \gamma, \end{align*}

where $K_1$ is a constant dependent only on $T,L,L_1, \ \textrm{and} \ p$ .

Proof of Theorem 5. Combining Lemmas 8 and 10 and using the Cauchy–Schwarz inequality, we have

\begin{align*} \mathbb{E}\left|x_{n,k+1}-x_{k+1}\right|^2 &\le 2 \, \mathbb{E}\left|x_{n,k+1}-\tilde{x}_{k+1}\right|^2 +2 \, \mathbb{E}\left|x_{k+1}-\tilde{x}_{k+1}\right|^2\\ &\le 2\tilde{C}^*_1\frac{\gamma^2}{m}+2\tilde{C}^*_2 \frac{k}{m} +2 K^{*}_1\frac{3^k\gamma}{m} . \end{align*}

This gives us

\begin{align*} \mathbb{E}\left|x_{n,k+1}-x_{k+1}\right|^2 & \le K_1 \frac{3^k\gamma}{m}, \end{align*}

where $K_1=2\tilde{C}^*_1 +2\tilde{C}^*_2\frac{1}{T}+2K^*_1$ . Note that $\gamma/3^k<1$ and $\frac{k}{3^k\gamma}=\frac{k^2}{3^k T}<\frac{1}{T}$ since $k^2<3^k$ for all $k \ge 1$ .

Now, we shall address the problem with only positive weights.

Lemma 11. Let $x_{n,k}$ and $\tilde x_k$ be as defined in (5) and (C1). Then, under Assumptions 1–4, with $w_{i,k} \ge 0$ for all $k\le K$ ,

\begin{align*} \mathbb{E}\left|x_{n,k+1}-\tilde x_{k+1}\right|^2 &\le D^*_1\,\frac{\gamma^2}{m}+D^*_2\,\frac{k}{m}, \end{align*}

where

\begin{align*} D^{*}_1&=\exp\left(1+2\,T\, \mathbb{E}(h_1(u))+T^2\, Var(h_1(u))\right)\, e^{\pi^2/6}\,\left\|\sigma(x_0)\right\|^2_F\\ D^*_2&=\exp\left(1+2\,T\, \mathbb{E}(h_1(u))+T^2\, Var(h_1(u))\right)\, e^{\pi^2/6}\,\left(T^2+1\right)\\&\quad \left(2 \, L^2_1\,p\,C^2_1 T^2 e^{2\,LT} + 2\sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\right). \end{align*}

Proof of Lemma 11. Recall the definitions of (5) and (C1). Then

\begin{align*} x_{n,k+1}-\tilde x_{k+1}&=x_{n,k}-\tilde x_{k}-\gamma \sum_{i=1}^{n}w_{i,k}\left(\nabla l\left(x_{n,k},u_{i,k}\right)-g(\tilde x_k)\right)\\ &=x_{n,k}-\tilde x_{k}-\gamma\sum_{i=1}^{n}w_{i,k}\left(\nabla l\left(x_{n,k},u_{i,k}\right)-\nabla l(\tilde{x}_k,u_{i,k}\right)\\&\quad-\gamma\sum_{i=1}^{n}\left(\nabla l(\tilde{x}_k,u_{i,k} -\gamma g(\tilde x_k)\right). \end{align*}

This implies that

\begin{align*} &\left| x_{n,k+1}-\tilde x_{k+1}\right|\\ & \le \left|x_{n,k}-\tilde x_{k}\right| +\gamma \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_i)-\nabla l(\tilde{x}_k,u_i)\right)\right|+\gamma \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(\tilde{x}_k,u_i)-\nabla g(\tilde{x}_k)\right)\right|\\ &\le \left(1+\gamma \sum_{i=1}^{n} w_i \, h(u_i)\right)\left|x_{n,k}-\tilde{x}_k\right|+\gamma \left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(\tilde{x}_k,u_i)-\nabla g(\tilde{x}_k)\right)\right|. \end{align*}

Therefore,

\begin{align*} &\mathbb{E}\left| x_{n,k+1}-\tilde x_{k+1}\right|^2\\ &\le \left(1+\frac{1}{k^2}\right)\left[1+2\,\gamma \, \mathbb{E}(h_1(u))+\gamma^2\, \frac{1}{m}\left(1-\frac{m}{n}\right)\textrm{Var}(h_1(u))\right]\mathbb{E}\left|x_{n,k}-\tilde{x}_k\right|^2\\&\quad +\gamma^2 \left(k^2+1\right)\frac{1}{m}\left\|\sigma(\tilde{x}_k)\right\|^2_F\\ & \le \left[1+2\,\gamma \, \mathbb{E}(h_1(u))+\gamma^2\, \frac{1}{m}\textrm{Var}(h_1(u))\right]^k\Bigg\{\left[\prod_{j=1}^{k}\left(1+\frac{1}{j^2}\right)\right]\mathbb{E}\left|x_1-\tilde{x}_1\right|^2\\ &\quad +\sum_{j=1}^{k-1}\prod_{i=1}^{j}\left(1+\frac{1}{\left(k-i+1\right)^2}\right)\frac{\gamma^2\left((k-j)^2+1\right)}{m}\left|\left|\sigma(\tilde{x}_{k-j})\right|\right|^2_F\Bigg\}\\ &\quad +\frac{\gamma^2\left(k^2+1\right)}{m}\left|\left|\sigma(\tilde x_k)\right|\right|^2_F. \end{align*}

Thus,

\begin{align*} &\mathbb{E}\left| x_{n,k+1}-\tilde x_{k+1}\right|^2\\ & \le \exp\left(1+2\,T\, \mathbb{E}(h_1(u))+T^2\, \textrm{Var}(h_1(u))\right)\, e^{\pi^2/6}\, \frac{\gamma^2}{m}\left\|\sigma(x_0)\right\|^2_F\\ & \quad +\exp\left(1+2\,T\, \mathbb{E}(h_1(u))+T^2\, \textrm{Var}(h_1(u))\right)\, e^{\pi^2/6}\,\left(T^2+1\right)\\ &\quad \quad \left(2 \, L^2_1\,p\,C^2_1 T^2 e^{2\,LT} + 2\sup\limits_{0\le t\le T} \left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\right)\, \frac{k}{m}\\ &=D^*_1 \frac{\gamma^2}{m}+D^*_2\frac{k}{m}. \end{align*}

Hence, the proof is complete.

Proposition 6. Suppose Assumptions 1–4 hold with $w_{i,k}\ge 0$ for all $k=1,2,\ldots,K$ . Recall $x_{n,k}$ and $x_k$ from (5) and (4). Then for any $k \le K= \left[T/\gamma\right]$ ,

\begin{align*} \mathbb{E}\left|x_{n,k}-x_{k}\right|^2 &\le K_{11}\cdot \frac{\gamma^2}{m}+K_{12}\frac{k}{m}, \end{align*}

where $K_{11}$ is a constant dependent only on $T,L,L_1, \ \textrm{and} \ p$ .

Proof. Combining Lemmas 8 and 11 and using the Cauchy–Schwarz inequality, we have

\begin{align*} \mathbb{E}\left|x_{n,k+1}-x_{k+1}\right|^2 &\le 2 \, \mathbb{E}\left|x_{n,k+1}-\tilde{x}_{k+1}\right|^2 +2 \, \mathbb{E}\left|x_{k+1}-\tilde{x}_{k+1}\right|^2\\ &\le 2\tilde{C}^*_1\frac{\gamma^2}{m}+2\tilde{C}^*_2 \frac{k}{m} +2\tilde{D}^*_1\frac{\gamma^2}{m}+2\tilde{D}^*_2 \frac{k}{m}. \end{align*}

In the last step, we use the fact that $m\ge K^2$ . Hence,

\begin{align*} K_{11}&=2\,\left(C^*_1+D^*_1\right)\\ K_{12}&=2\,\left(C^*_2+D^*_2\right). \end{align*}

Now we present a few lemmas, which shall be very important for our next steps.

Lemma 12. Under Assumptions 1–4,

\begin{align*} \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2 \le \tilde{C_3}+\tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}, \end{align*}

where $\tilde C_3=4L^2C_1^2T^2e^{2LT}+4\sup\limits_{0\le t\le T}\left|g(\tilde{X}_t)\right|^2$ and $\tilde{C}_1,\tilde{C}_2$ are $2\max(pL^2_1,L^2)\, \tilde{C}^*_1$ and $2\max(pL^2_1,L^2)\, \tilde{C}^*_2$ , respectively, where $\tilde{C}^*_1,\tilde{C}^*_2$ are as defined in Lemma 8. In addition,

\begin{align*} \mathbb{E}\left({Tr} \sigma^2(D_{k\gamma})\right) \le \tilde{C_4}+\tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}, \end{align*}

where $\tilde{C}_4=4L^2C_1^2T^2e^{2LT}+4\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F$ .

Proof. Note that

\begin{align*} \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2 &=\mathbb{E}\left|\nabla g(x_k)\right|^2\\ &\le 2L^2 \ \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+4L^2\left|\tilde{x}_k-\tilde{X}_{k\gamma}\right|^2+4\left|\nabla g(\tilde X_{k\gamma})\right|^2. \end{align*}

Using Lemmas 8 and 6, we find that

\begin{align*} \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2&\le 4L^2C_1^2 k^2\gamma^2\left(1+L\gamma\right)^{2k}+4\sup\limits_{0\le t\le T}\left|g(\tilde{X}_t)\right|^2\\ &\quad + \tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}. \end{align*}

This implies

\begin{align*} \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2 &\le 4L^2C_1^2T^2e^{2LT}+4\sup\limits_{0\le t\le T}\left|g(\tilde{X}_t)\right|^2\\[4pt] &\quad + \tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}\\[4pt] &=\tilde{C_3}+\tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}, \end{align*}

where $\tilde C_3=4L^2C_1^2T^2e^{2LT}+4\sup\limits_{0\le t\le T}\left|g(\tilde{X}_t)\right|^2$ and $\tilde{C}_1,\tilde{C}_2$ are $2\max(pL^2_1,L^2)\, \tilde{C}^*_1$ and $2\max(pL^2_1,L^2)\, \tilde{C}^*_2$ , respectively, where $\tilde{C}^*_1,\tilde{C}^*_2$ are as defined in Lemma 8. We now do the same with $\sigma(\! \cdot \!)$ . In fact,

\begin{align*} \mathbb{E}\left(\textrm{Tr} \sigma^2(D_{k\gamma})\right)&=\mathbb{E}\left|\left|\sigma(x_k)\right|\right|^2_F\\[4pt] & \ \le 2\, p\,L^2_1 \ \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+2 \left|\left|\sigma(\tilde{x}_k)\right|\right|^2_F\\[4pt] & \ \le 2\,p\,L^2_1 \ \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+4 p L^2_1C_1^2 k^2 \gamma^2\left(1+L\gamma\right)^{2k}+4\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F. \end{align*}

Using this fact, and the bounds from Lemmas 8 to 6, we have

\begin{align*} \mathbb{E}\left(\textrm{Tr} \sigma^2(D_{k\gamma})\right) & \le 4pL^2_1C_1^2 k^2\gamma^2\left(1+L\gamma\right)^{2k}+4\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\\[4pt] &\quad +\tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m} \\[4pt] &\le 4pL_1^2C_1^2T^2e^{2LT}+4\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\\[4pt] &\quad + \tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}\\[4pt] &\le \tilde{C}_4 + \tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}, \end{align*}

where $\tilde{C}_4=4pL_1^2C_1^2T^2e^{2LT}+4\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F$ and $\tilde{C}_1,\tilde{C}_2$ are $2\max(pL^2_1,L^2)\, \tilde{C}^*_1$ and $2\max(pL^2_1,L^2)\, \tilde{C}^*_2$ , respectively, where $\tilde{C}^*_1,\tilde{C}^*_2$ are as defined in Lemma 8.

Lemma 13. Under Assumptions 1–4, for $t \in (k\gamma,(k+1)\gamma]$ , we have

\begin{align*} \int_{k\gamma}^{t} \,\mathbb{E}\left|D_{s}-D_{k\gamma}\right|^2 \le K_{11}\gamma^3 + K_{12}\frac{\gamma^2}{m} \end{align*}

where $K_{11}, K_{12}$ are functions dependent on $L,L_1,T$ , and p.

Proof of Lemma 13. Using the definition of $D_{s}$ , we have

\begin{align*} \mathbb{E}\left|D_{s}-D_{k\gamma}\right|^2&= \,\mathbb{E}{\left|-\nabla g(D_{k\gamma})\left(s-k\gamma\right)+\sqrt{\frac{\gamma}{m}}\sigma(D_{k\gamma})\left(B_s-B_{k\gamma}\right)\right|}^2\\[5pt] &\le 2 \,\mathbb{E}{\left[\left|\nabla g(D_{k\gamma})\right|\left(s-k\gamma\right)\right]}^2+2\frac{\gamma}{m} \,\mathbb{E}{\left|\int_{s}^{k\gamma}\sigma(D_{k\gamma}) \,\textrm{d}B_l\right|}^2\\[5pt] &=2\left(s-k\gamma\right)^2 \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2+2\frac{\gamma}{m} \,\mathbb{E}\left(\textrm{Tr} \sigma^2(D_{k\gamma})\right)\left(s-k\gamma\right). \end{align*}

Hence, for $t \in [k\gamma,(k+1)\gamma]$ ,

\begin{align*} \int_{k\gamma}^{t} \mathbb{E}\left|D_{s}-D_{k\gamma}\right|\le 2\int_{k\gamma}^{t} \left(s-k\gamma\right)^2 \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2+2\frac{\gamma}{m}\int_{k\gamma}^{t}\mathbb{E}\left|\left|\sigma(D_{k\gamma})\right|\right|^2_F\left(s-k\gamma\right). \end{align*}

Now, using Lemma 12 the first term is bounded by

\begin{align*} 2\int_{k\gamma}^{t} \left(s-k\gamma\right)^2 \mathbb{E}\left|\nabla g(D_{k\gamma})\right|^2 &\le 2\int_{k\gamma}^{t}\left(s-k\gamma\right)^2 \left(\tilde{C}_3+\tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}\right)\\[5pt] &\le \frac{2\gamma^3}{3}\tilde{C}_3+\frac{2}{3}\tilde{C}_1\frac{\gamma^5}{m}+\frac{2}{3}\tilde{C}_2 \frac{\gamma^2 T}{m}. \end{align*}

For the second term we can do the exact same thing. In fact,

\begin{align*} 2\frac{\gamma}{m}\int_{k\gamma}^{t}\mathbb{E}\left|\left|\sigma(D_{k\gamma})\right|\right|^2_F\left(s-k\gamma\right)&\le 2 \int_{k\gamma}^{t}\frac{\gamma}{m}\left(s-k\gamma\right)\left(\tilde{C}_4+\tilde{C}_1 \frac{\gamma^2}{m}+\tilde{C}_2 \frac{k}{m}\right). \end{align*}

Combining the above terms we get

\begin{align*} \int_{k\gamma}^{t} \mathbb{E}\left|D_{s}-D_{k\gamma}\right|^2 &\le \tilde K_{11}\gamma^3+\tilde{K}_{12} \frac{\gamma^5}{m}+\tilde{K}_{13}\frac{\gamma^2}{m}, \end{align*}

where

\begin{align*} \tilde{K}_{11}=\left(\frac{2}{3}\tilde{C}_3+\frac{1}{m}\tilde{C}_4\right)\!, \end{align*}

and $K_{12}$ can be taken as

\begin{align*} \tilde{K}_{12}=\frac{5}{3}\tilde{C}_1, \end{align*}

and

\begin{align*} \tilde{K}_{13}= \frac{5T}{3}\tilde{C}_2. \end{align*}

Hence, we can write

\begin{align*} \int_{k\gamma}^{t} \mathbb{E}\left|D_{s}-D_{k\gamma}\right|^2 \le K_{11}\gamma^3 + K_{12}\frac{\gamma^2}{m}, \end{align*}

where $K_{11}=\tilde{K}_{11}$ , $K_{12}=\tilde{K}_{12}+\tilde{K}_{13}$ . This completes the proof.

Next we prove Theorem 3.

Proof of Theorem 3. We focus on the interval $(k\gamma,(k+1)\gamma]$ , i.e. $t \in (k\gamma,(k+1)\gamma]$ . With some abuse of notation we define $j\gamma$ for $j=0,1,2,\ldots,k-1$ ; with $jk=t$ . We use this abuse of notation as this helps with otherwise cumbersome notation. Using the definition of (6) and (7), we have

\begin{align*} \left|X_t-D_{t}\right|&\le \left|\int_{0}^{t}\nabla g(X_s)-\int_{0}^{t}\sum_{j=0}^{k-1}\nabla g(D_{j\gamma})I(s)_{[j\gamma,(j+1)\gamma)}\right|\,\textrm{d}s\\ &+\sqrt{\frac{\gamma}{m}}\left|\int_{0}^{t}\left(\sigma(X_s)-\sum_{j=0}^{k-1}\sigma(D_{j\gamma})I(s)_{[j\gamma,(j+1)\gamma)}\right)\,\textrm{d}B_s\right|. \end{align*}

Thus,

\begin{align*} &\left|X_t-D_{t}\right|\\ &\le \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\left|\nabla g(X_s)-\nabla g(D_{j\gamma})\right| \,\textrm{d}s+\sqrt{\frac{\gamma}{m}}\left|\sum_{j=0}^{k-1} \int_{j\gamma}^{(j+1)\gamma}\left(\sigma(X_s)-\sigma(D_{j\gamma})\right)\,\textrm{d}B_s\right| \\ &\le L \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\left| X_s-D_{j\gamma}\right|+\sqrt{\frac{\gamma}{m}}\left|\sum_{j=0}^{k-1} \int_{j\gamma}^{(j+1)\gamma}\left(\sigma(X_s)-\sigma(D_{j\gamma})\right)\,\textrm{d}B_s\right|. \end{align*}

Hence, by using triangle inequality,

\begin{align*} &\left|X_t-D_{t}\right|\\ &\le L \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\left| X_s-D_{s}\right|+L \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\left| D_{s} - D_{j\gamma}\right|\\ & \quad + \sqrt{\frac{\gamma}{m}} \left|\sum_{j=0}^{k-1} \int_{j\gamma}^{(j+1)\gamma}\left(\sigma(X_s)-\sigma(D_{s})\right)\,\textrm{d}B_s\right|+\left|\sum_{j=0}^{k-1} \int_{j\gamma}^{(j+1)\gamma}\left(\sigma(D_{s})-\sigma(D_{j\gamma})\right)\,\textrm{d}B_s\right|\\ &\le L \int_{0}^{t}\left| X_s-D_{s}\right|+ L \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\left| D_{s} - D_{j\gamma}\right|+\sqrt{\frac{\gamma}{m}} \left| \int_{0}^{t}\left(\sigma(X_s)-\sigma(D_{s})\right)\,\textrm{d}B_s\right|\\ &\quad +\sqrt{\frac{\gamma}{m}} \sum_{j=0}^{k-1} \left| \int_{j\gamma}^{(j+1)\gamma}\left(\sigma(X_s)-\sigma(D_{j\gamma})\right)\,\textrm{d}B_s\right|. \end{align*}

Now squaring both sides and applying the Cauchy–Schwarz inequality first on all the terms and then the first two integrals (on the second term we apply the Cauchy–Schwarz inequality twice for the sum and then for the integral), we get

\begin{align*} &\left|X_t-D_{t}\right|^2\\ &\le 4L^2 t\int_{0}^{t}\left| X_s-D_{s}\right|^2+ 4L^2 k\gamma \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\left|D_{s}-D_{j\gamma}\right|^2\\ &\quad+ 4\frac{\gamma}{m}\left|\int_{0}^{t}\left(\sigma(X_s)-\sigma(D_{s})\right)\,\textrm{d}B_s\right|^2 +4\frac{\gamma}{m}k\sum_{j=0}^{k-1}\left|\int_{j\gamma}^{(j+1)\gamma}\left(\sigma(D_{s})-\sigma(D_{j\gamma})\right)\,\textrm{d}B_s\right|^2 . \end{align*}

Taking expectation,

\begin{align*} \mathbb{E}\left|X_t-D_{t}\right|^2 &\le 4L^2 t\int_{0}^{t}\mathbb{E}\left| X_s-D_{s}\right|^2+ 4L^2 k\gamma \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{j\gamma}\right|^2\\[2pt] &\quad+ 4\frac{\gamma}{m} \,\mathbb{E}\left|\int_{0}^{t}\left(\sigma(X_s)-\sigma(D_{s})\right)\,\textrm{d}B_s\right|^2\\[2pt] &\quad +4\frac{\gamma}{m}k\sum_{j=0}^{k-1}\mathbb{E}\left|\int_{j\gamma}^{(j+1)\gamma}\left(\sigma(D_{s})-\sigma(D_{j\gamma})\right)\,\textrm{d}B_s\right|^2 . \end{align*}

Use the Itô isometry on the last two expressions to give us

\begin{align*} &\mathbb{E}\left|X_t-D_{t}\right|^2 \\[2pt] &\le 4L^2 t\int_{0}^{t}\mathbb{E}\left| X_s-D_{s}\right|^2+ 4L^2 k\gamma \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{j\gamma}\right|^2\\[2pt] &\quad+ 4\frac{\gamma}{m}\mathbb{E}\int_{0}^{t}\left|\left|\sigma(X_s)-\sigma(D_{s})\right|\right|_F^2\,\textrm{d}s +4\frac{k\gamma}{m}\sum_{j=0}^{k-1}\mathbb{E}\int_{j\gamma}^{(j+1)\gamma}\left|\left|\sigma(D_{s})-\sigma(D_{j\gamma})\right|\right|_F^2\,\textrm{d}s. \end{align*}

Since $\sigma$ is Lipschitz, we obtain

\begin{align*} \mathbb{E}\left|X_t-D_{t}\right|^2 &\le 4L^2 t\int_{0}^{t}\mathbb{E}\left| X_s-D_{s}\right|^2+ 4L^2 k\gamma \sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{j\gamma}\right|^2\\[2pt] &\quad+ 4\frac{\gamma}{m}pL_1^2\int_{0}^{t}\mathbb{E}\left|X_s-D_{s}\right|^2\,\textrm{d}s +4\frac{k\gamma}{m}pL_1^2 \sum_{j=0}^{k-1} \int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{k\gamma}\right|^2\,\textrm{d}s\\[2pt] &=\left(4L^2t + 4\frac{\gamma}{m}pL_1^2\right)\int_{0}^{t}\mathbb{E}\left| X_s-D_{s}\right|^2\\[2pt] &\quad \quad \quad +\left(4L^2 k\gamma+4\frac{k\gamma}{m}pL_1^2\right)\sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{j\gamma}\right|^2. \end{align*}

Now, from the last step we apply the Gronwall inequality to get

\begin{align*} &\mathbb{E}\left|X_t-D_{t}\right|^2\\ &\le \left[\left(4L^2 k\gamma+4\frac{k\gamma}{m}pL_1^2\right)\sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{j\gamma}\right|^2\right]\cdot\exp\left(4L^2\,T^2 + 4\frac{\gamma}{m}pL_1^2 T\right). \end{align*}

By using the fact $\gamma K=T$ , we have

\begin{align*} &\mathbb{E}\left|X_t-D_{t}\right|^2\\ &\le \left[\left(4L^2 T+4\frac{T}{m}pL_1^2\right)\sum_{j=0}^{k-1}\int_{j\gamma}^{(j+1)\gamma}\mathbb{E}\left|D_{s}-D_{j\gamma}\right|^2\right]\cdot\exp\left(4L^2T^2 + 4\frac{\gamma}{m}pL_1^2 T\right). \end{align*}

Invoking Lemma 13, one has

\begin{align*} &\mathbb{E}\left|X_t-D_{t}\right|^2\\ &\le \left[\left(4L^2 T+4\frac{T}{m}pL_1^2\right)\sum_{j=0}^{k-1}\left(K_{11}\gamma^3+K_{12} \frac{\gamma^2}{m} \right)\right]\cdot\exp\left(4L^2T^2 + 4\frac{\gamma}{m}pL_1^2 T\right)\\ &\le \left[\left(4L^2 T+4\frac{T}{m}pL_1^2\right)\left(K_{11} T\gamma^2+K_{12}\frac{T\gamma}{m} \right)\right]\cdot\exp\left(4L^2T^2 + 4\frac{\gamma}{m}pL_1^2 T\right)\\ &= C_{11}\gamma^2+C_{12}\frac{\gamma}{m}, \end{align*}

where

\begin{align*} C_{11}=\left(4L^2 T+4\frac{T}{m}pL_1^2\right)\exp\left(4L^2T^2 + 4\frac{\gamma}{m}pL_1^2 T\right)T K_{11} \end{align*}

and

\begin{align*} C_{12}=\left(4L^2 T+4\frac{T}{m}pL_1^2\right)\exp\left(4L^2T^2 + 4\frac{\gamma}{m}pL_1^2 T\right)T K_{12}. \end{align*}

Hence,

\begin{align*} W^2_2(X_t,D_{t})\le C_{11}\gamma^2+C_{12}\frac{\gamma}{m}. \end{align*}

We complete the proof.

We now prove another lemma before proceeding to one of the main theorems.

Lemma 14. Under Assumptions 1–4,

\begin{align*} &\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\nabla l(Y_{n,k\gamma},u_{i,k})-\sum_{i=1}^{n}w_{i,k}\nabla l(D_{k\gamma},u_{i,k})\right|^2\\ &\le \left[\frac{1}{m}\left(\mathbb{E}\left(h_1^2(U)\right)-L^2\right)+L^2\right]\mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2 . \end{align*}

Proof of Lemma 14. Again we start by bounding the second moment

\begin{align*} & \mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\nabla l(Y_{n,k\gamma},u_{i,k})-\sum_{i=1}^{n}w_{i,k}\nabla l(D_{k\gamma},u_{i,k})\right|^2\\[-0.5pt] &=\mathbb{E}\sum_{i=1}^{n}w^2_{i,k}\left|\nabla l(Y_{n,k\gamma},u_{i,k})-l(D_{k\gamma},u_{i,k})\right|^2\\[-0.5pt] &\quad + \,\mathbb{E}\sum_{i,j,i\ne j} w_{i,k}w_{j,k}\left(\nabla l(Y_{n,k\gamma},u_{i,k})-\nabla l(D_{k\gamma},u_{i,k})\right)^{\mathsf{T}}\\[-0.5pt] &\quad \quad \quad \quad \quad \quad \quad \quad \quad \left(\nabla l(Y_{n,k\gamma},u_{j,k})-\nabla l(D_{k\gamma},u_{j,k})\right) . \end{align*}

Hence,

\begin{align*} & \mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\nabla l(Y_{n,k\gamma},u_{i,k})-\sum_{i=1}^{n}w_{i,k}\nabla l(D_{k\gamma},u_{i,k})\right|^2\\[-0.5pt] & \quad \le \frac{1}{m} \ \mathbb{E}\left(h_1^2(u)\right)\mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2\\[-0.5pt] &\quad \quad + \sum_{i,j,i\ne j} \frac{m-1}{mn(n-1)} \,\mathbb{E}\left|\nabla g(Y_{n,k\gamma})-\nabla g(D_{k\gamma})\right|^2\\[-0.5pt] &\quad \le \frac{1}{m} \ \mathbb{E}\left(h_1^2(u)\right)\mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2 \\[-0.5pt] &\quad \quad + L^2\left(1-\frac{1}{m}\right) \mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2 . \end{align*}

This completes the proof.

Next we exhibit that the interpolated M-SGD process and the interpolated SGD with scaled normal error are close. Here $t \in (k\gamma,(k+1)\gamma]$ .

Proposition 7. Under Assumptions 1–4, for $t\in (k\gamma,(k+1)\gamma]$ , we have

\begin{align*} W_2^2(Y_{n,t},D_{t})\le \tilde{J}_1\frac{3^k\gamma}{m}, \end{align*}

where $\tilde{J}_1$ is a constant dependent on $T,L,L_1,p,$ and $\mathbb{E}\left(h_1^2(u)\right)$ .

Proof of Proposition 7. Using the definitions of (6) and (8), we have

\begin{align*} \left|Y_{n,t}-D_{t}\right|&\le \left|Y_{n,k\gamma}-D_{k\gamma}\right| +\left(t-k\gamma\right)\left|\sum_{i=1}^{n}w_{i,k}\nabla l(Y_{n,k\gamma},u_{i,k})-\nabla g(D_{k\gamma})\right|\\[-0.5pt] &\quad +\sqrt{\frac{\gamma}{m}}\left|\sigma(D_{k\gamma})\left(B_t-B_{k\gamma}\right)\right|\\[-0.5pt] &\le \left|Y_{n,k\gamma}-D_{k\gamma}\right| \ +\left(t-k\gamma\right)\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(Y_{n,k\gamma},u_{i,k})-\nabla l(D_{k\gamma},u_{i,k})\right)\right|\\[-0.5pt] &\quad +\left(t-k\gamma\right)\left|\sum_{i=1}^{n}w_{i,k}\nabla l(D_{k\gamma},u_{i,k})-\nabla g(D_{k\gamma})\right| \\[-0.5pt] &\quad \quad \quad +\sqrt{\frac{\gamma}{m}}\left|\sigma(D_{k\gamma})\left(B_t-B_{k\gamma}\right)\right|. \end{align*}

Thus, we get

\begin{align*} &\left|Y_{n,t}-D_{t}\right|^2\\ &\le 4 \left|Y_{n,k\gamma}-D_{k\gamma}\right|^2 +4\left(t-k\gamma\right)^2\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(Y_{n,k\gamma},u_{i,k})-\nabla l(D_{k\gamma},u_{i,k})\right)\right|^2\\ &\quad +4\left(t-k\gamma\right)^2\left|\sum_{i=1}^{n}w_{i,k}\nabla l(D_{k\gamma},u_{i,k})-\nabla g(D_{k\gamma})\right|^2 +4\frac{\gamma}{m}\left|\sigma(D_{k\gamma})\left(B_t-B_{k\gamma}\right)\right|^2. \end{align*}

Taking expectation, we see

\begin{align*} &\mathbb{E}\left|Y_{n,t}-D_{t}\right|^2\\ &\le 4 \, \mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2 +4\left(t-k\gamma\right)^2\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\left(\nabla l(Y_{n,k\gamma},u_{i,k})-\nabla l(D_{k\gamma},u_{i,k})\right)\right|^2\\ &\quad +4\left(t-k\gamma\right)^2\mathbb{E}\left|\sum_{i=1}^{n}w_{i,k}\nabla l(D_{k\gamma},u_{i,k})-\nabla g(D_{k\gamma})\right|^2 +4\frac{\gamma}{m}\mathbb{E}\left|\sigma(D_{k\gamma})\left(B_t-B_{k\gamma}\right)\right|^2. \end{align*}

This implies

\begin{align*} &\mathbb{E}\left|Y_{n,t}-D_{t}\right|^2\\ &\le 4 \,\mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2 \ +4\left(t-k\gamma\right)^2\left(\frac{1}{m}\left(\mathbb{E}\left(h_1^2(u)\right)-L^2\right)+L^2\right)\mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2\\ &\quad +4\left(t-k\gamma\right)^2\frac{1}{m} \,\mathbb{E}\left|\left|\sigma(D_{k\gamma})\right|\right|^2_F\ +4\frac{\gamma}{m}\left(t-k\gamma\right)\mathbb{E}\left|\left|\sigma(D_{k\gamma})\right|\right|^2_F. \end{align*}

The last line follows using Lemma 14 and the Itô isometry. Therefore,

\begin{align*} \mathbb{E}\left|Y_{n,t}-D_{t}\right|^2&\le \left[4+4\left(t-k\gamma\right)\left(\frac{1}{m}\left(\mathbb{E}\left(h_1^2(u)\right)-L^2\right)+L^2\right)\right]\mathbb{E}\left|x_{n,k}-x_k\right|^2\\ &\quad+2\left[\frac{1}{m}4\left(t-k\gamma\right)^2+4\frac{\gamma}{m}\left(t-k\gamma\right)\right]\left(pL^2_1 \, \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+\left|\left|\sigma(\tilde{x}_k)\right|\right|^2_F\right). \end{align*}

Here we use the fact that $\mathbb{E}\left|Y_{n,k\gamma}-D_{k\gamma}\right|^2= \,\mathbb{E}\left|x_{n,k}-x_k\right|^2$ . We also use the facts $\mathbb{E}\left|D_{k\gamma}-\tilde{x}_k\right|^2= \,\mathbb{E}\left|x_{k}-\tilde{x}_k\right|^2$ and $\sigma(\! \cdot \!)$ is Lipschitz. Using this, along with Proposition 5 and Lemmas 8 and 12, we obtain

\begin{align*} \mathbb{E}\left|Y_{n,t}-D_{t}\right|^2 \le J_{1}+J_{2}, \end{align*}

where

\begin{align*} J_{1}&=\left\{4+4\left(t-k\gamma\right)\left(\frac{1}{m}\left[\mathbb{E}\left(h_1^2(u)\right)-L^2\right]+L^2\right)\right\}\\ &\quad \quad \cdot K_1 \frac{3^k\gamma}{m}, \end{align*}

and

\begin{align*} J_2&=2\left[\frac{1}{m}4\left(t-k\gamma\right)^2+4\frac{\gamma}{m}\left(t-k\gamma\right)\right]\\ &\quad \quad \cdot \Bigg\{\tilde{C}_1\frac{\gamma^2}{m}+\tilde{C}_2\frac{k}{m}+ \left[2C_1^2 k^2 \gamma^2L_1^2p\left(1+L\gamma\right)^{2k}+2\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\right]\Bigg\}\\ &\le2\left[\frac{1}{m}4\gamma^2+4\frac{\gamma}{m}\left(t-k\gamma\right)\right]\\ &\quad \quad \cdot \Bigg\{\tilde{C}_1\frac{\gamma^2}{m}+\tilde{C}_2\frac{k}{m}+ \left[2C_1^2 T^2 L_1^2p e^{2TL}+2\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\right]\Bigg\}. \end{align*}

Note that $J_1 \le \frac{3^k\gamma}{m} J_{11}$ , where

\begin{align*} J_{11}&= K_1 \left[4+4\,\mathbb{E}\left(h_1^2(u)\right)\right]. \end{align*}

In addition,

\begin{align*} J_2=J_{21} \frac{\gamma^4}{m^2}+J_{22}\frac{\gamma}{m^2}+J_{23}\frac{\gamma^2}{m} \end{align*}

where $J_{21}$ , $J_{22}$ , and $J_{23}$ can be chosen as

\begin{align*} J_{21}&=8 \tilde{C}_1,\\ J_{22}&=8 \tilde{C}_2 T,\\ J_{23}&=8 \left[2C_1^2 T^2 L_1^2p e^{2TL}+2\sup\limits_{0\le t\le T}\left|\left|\sigma(\tilde{X}_t)\right|\right|^2_F\right]. \end{align*}

We conclude

\begin{align*} E\left|Y_{n,t}-D_{t}\right|^2&\le \frac{3^k\gamma}{m} J_{11}+J_{21} \frac{\gamma^4}{m^2}+J_{22}\frac{\gamma}{m^2}+J_{23}\frac{\gamma^2}{m} \\ &\le \tilde J_{1} \frac{3^k\gamma}{m}, \end{align*}

where we can consider $\tilde{J}_1$ as

\begin{align*} \tilde{J}_1 = J_{11}+J_{21}+J_{22}+J_{23}. \end{align*}

Therefore, it follows that

\begin{align*} W_2^2(Y_{n,t},D_{t})\le \tilde{J}_1\frac{3^k\gamma}{m}. \end{align*}

The proof is finished.

Next we prove one of our main theorems.

Proof of Theorem 4. Let $t \in [k\gamma,(k+1)\gamma)$ . Using the fact that the Wasserstein distance exhibits the inequality $W^2_2(\mu,\nu)\le 2 W^2_2(\mu,P)+2W^2_2(P,\nu)$ (where $\mu$ , $\nu$ , and P are probability measures),

\begin{align*} W_2^2(Y_{n,t},X_t)& \le 2W_2^2(Y_{n,t},D_{t})+2W_2^2(D_{t},X_t)\\ &\le 2\tilde{J}_1\frac{3^k\gamma}{m}+2C_{11}\gamma^2+2C_{12}\frac{\gamma}{m}\\ &\le C_{21}\gamma^2+C_{22}\frac{3^k\gamma}{m}, \end{align*}

where $C_{21}=2C_{11}$ and $C_{22}=2\tilde{J}_1+C_{12}$ . Hence, we conclude the proof.

Proposition 8. Under Assumptions 1–4, with $w_{ik} \ge 0$ , for $t\in (k\gamma,(k+1)\gamma]$ ,

\begin{align*} W_2^2(Y_{n,t},D_{t})\le \tilde{J}_{11}\frac{\gamma^2}{m}+\tilde{J}_{12}\frac{\gamma}{m^2}+\tilde{J}_{13}\frac{k}{m} \end{align*}

where $\tilde{J}_{11}, J_{12}$ are constants dependent on $T,L,L_1,p,$ and $\mathbb{E}\left(h_1^2(u)\right)$ .

Proof of Proposition 8. From Proposition 7, we know that

\begin{align*} \mathbb{E}\left|Y_{n,t}-D_{t}\right|^2&\le \left[4+4\left(t-k\gamma\right)\left(\frac{1}{m}\left(\mathbb{E}\left(h_1^2(u)\right)-L^2\right)+L^2\right)\right]\mathbb{E}\left|x_{n,k}-x_k\right|^2\\ &\quad+2\left[\frac{1}{m}4\left(t-k\gamma\right)^2+4\frac{\gamma}{m}\left(t-k\gamma\right)\right]\left(pL^2_1 \, \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+\left|\left|\sigma(\tilde{x}_k)\right|\right|^2_F\right). \end{align*}

For the first term, we get

\begin{align*} &\left[4+4\left(t-k\gamma\right)\left(\frac{1}{m}\left(\mathbb{E}\left(h_1^2(u)\right)-L^2\right)+L^2\right)\right]\mathbb{E}\left|x_{n,k}-x_k\right|^2\\ &\le 4\,\left(1+\mathbb{E}(h^2_1(u))\right)\left(K_{11}\frac{\gamma^2}{m}+K_{12}\frac{k}{m}\right). \end{align*}

For the second term, we get

\begin{align*} &2\left[\frac{1}{m}4\left(t-k\gamma\right)^2+4\frac{\gamma}{m}\left(t-k\gamma\right)\right]\left(pL^2_1 \, \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+\left|\left|\sigma(\tilde{x}_k)\right|\right|^2_F\right)\\ &\le 16\, \frac{\gamma^2}{m}\, \left[p\, L^2_1\left(C^*_1\frac{\gamma^2}{m}+C^*_2\frac{k}{m}\right)+\left\|\sigma(\tilde{x}_k)\right\|^2_F\right]\\ &=16\, p\, L^2_1\, C^*_1 \frac{\gamma^4}{m^2}+16\, p\, L^2_1\, C^*_2\, T \frac{\gamma}{m^2}+16\, \left\|\sigma(\tilde{x}_k)\right\|^2_F\frac{\gamma^2}{m}. \end{align*}

Therefore, we have

\begin{align*} &\mathbb{E}\left|Y_{n,t}-D_{t}\right|^2 \\ & \le \left[4\, K_{11}\big(1+\mathbb{E}(h^2_1(u))\big)+16\, p\, L^2_1\,C^*_1+16\, \left\|\sigma(\tilde{x}_k)\right\|^2_F\right]\frac{\gamma^2}{m}\\&\quad+16\, p\, L^2_1\, C^*_2\, T \frac{\gamma}{m^2}+4\, K_{12}\big(1+\mathbb{E}(h^2_1(u))\big)\frac{k}{m}. \end{align*}

Hence, we are done with

\begin{align*} \tilde{J}_{11}&=\left[4\, K_{11}\left(1+\mathbb{E}(h^2_1(u))\right)+16\, p\, L^2_1\,C^*_1+16\, \left\|\sigma(\tilde{x}_k)\right\|^2_F\right]\\ \tilde{J}_{12}&=16\, p\, L^2_1\, C^*_2\, T\\ \tilde{J}_{13}&=4\, K_{12}\left(1+\mathbb{E}(h^2_1(u))\right).\end{align*}

Proof of Theorem 5. We have

\begin{align*} W_2^2(Y_{n,t},X_t)& \le 2W_2^2(Y_{n,t},D_{t})+2W_2^2(D_{t},X_t)\\ &\le \tilde{J}_{11}\frac{\gamma^2}{m}+\tilde{J}_{12}\frac{\gamma}{m^2}+\tilde{J}_{13}\frac{k}{m}+2C_{11}\gamma^2+2C_{12}\frac{\gamma}{m}\\ &\le C_{23}\gamma^2+C_{24}\frac{\gamma}{m}. \end{align*}

Here

\begin{align*} C_{23}&=2\,C_{11}+\tilde{J}_{11}\\ C_{24}&=\tilde{J}_{12}+\frac{1}{T}\,\tilde{J}_{13}+2\,C_{12}.\end{align*}

C.1. Proofs for the convex regime

In the case of the objective function g being strongly convex, we derive bounds for the M-SGD algorithm with the structure as mentioned previously.

Consider (1). Under Assumptions 1–4 and 7, we find that the algorithm converges to the global optimum on average.

Proof of Proposition 3. We know

\begin{align*} x_{n,k+1}&=x_{n,k}-\gamma \sum_{i=1}^{n}w_{i,k}\nabla l(x_{n,k},u_{i,k})\\ &=x_{n,k}-\gamma \nabla g(x_{n,k})-\gamma \sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k})\right). \end{align*}

Hence, we have

\begin{align*} g(x_{n,k+1})&=g(x_{n,k}-\gamma \nabla g(x_{n,k})-\gamma \sum_{i=1}^{n}w_{i,k}(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k}))). \end{align*}

Therefore,

\begin{align*} g(x_{n,k+1})&=g(x_{n,k})-\gamma \nabla g(x_{n,k})^{\mathsf{T}}\left(\nabla g(x_{n,k})+\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k})\right)\right)\\ &\quad +\frac{\gamma^2}{2}\left(\nabla g(x_{n,k})+\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k})\right)\right)^{\mathsf{T}}\nabla^2g(\hat{x}_{n,k})\\ &\quad \quad \quad \ \left(\nabla g(x_{n,k})+\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k})\right)\right). \end{align*}

Thus,

\begin{align*} &g(x_{n,k+1}) \\&\le g(x_{n,k})-\gamma |\nabla g(x_{n,k})|^2-\gamma \nabla g(x_{n,k})^{\mathsf{T}}\left(\sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k})\right)\right)\\ &\quad +\frac{L\gamma^2}{2}\left|\nabla g(x_{n,k})+ \sum_{i=1}^{n}w_{i,k}\left(\nabla l(x_{n,k},u_{i,k})-\nabla g(x_{n,k})\right)\right|^2. \end{align*}

The last line follows from the fact that $\nabla g$ is Lipschitz. By taking expectation, we get

\begin{align*} &\mathbb{E}\left( g(x_{n,k+1})\right)\\& \le \mathbb{E} \left(g(x_{n,k})\right)-\gamma \,\mathbb{E}\left|\nabla g(x_{n,k})\right|^2+\frac{L\gamma^2}{2} \,\mathbb{E}\left|\nabla g(x_{n,k})\right|^2+\frac{L\gamma^2}{2m} \,\mathbb{E} \left(\textrm{Tr} \sigma^2(x_{n,k})\right) \\ &\le \mathbb{E}\left(g(x_{n,k})\right)-\gamma \left(1-\frac{L\gamma}{2}\right) \mathbb{E}\left|\nabla g(x_{n,k})\right|^2+\frac{L\gamma^2}{2m} \,\mathbb{E} \left(\textrm{Tr} \sigma^2(x_{n,k})\right) . \end{align*}

The second line follows as this is the online version of the algorithm, i.e. we refresh the $u_i$ at each iteration and $\nabla l(\cdot,u)$ is unbiased for $g(\! \cdot \!)$ . In addition, we use the definition that $\sigma(\! \cdot \!)$ is the variance of $\nabla l(\cdot,u)$ . From [Reference Boyd, Boyd and Vandenberghe5], we have

\[\left|\nabla g(x)\right|^2 \ge 2\lambda \left(g(x)-g(x^*)\right)\!, \ \forall x .\]

From the previous inequality, it follows that

\begin{align*} \mathbb{E} \left(g(x_{n,k+1})\right) &\le \mathbb{E} \left(g(x_{n,k})\right)-\gamma \lambda \left(2-L\gamma\right) \mathbb{E}\left(g(x_{n,k})-g(x^*)\right)+\frac{L\gamma^2}{2m} \,\mathbb{E} \left|\left|\sigma(x_{n,k})\right|\right|^2_F\\ &\le \mathbb{E} \left(g(x_{n,k})\right)-\gamma \lambda \left(2-L\gamma\right) \mathbb{E}\left(g(x_{n,k})-g(x^*)\right)+\frac{L\gamma^2}{2m}\{2 \, \mathbb{E} \left|\left|\sigma(x^*)\right|\right|^2_F\\ &\quad \quad \quad + 2pL^2_1 \,\mathbb{E}\left|x_{n,k}-x^*\right|^2\}. \end{align*}

The last line uses the fact that $\sigma(\! \cdot \!)$ is $\sqrt p L_1$ Lipschitz in the Frobenius norm, which follows from the fact that it is $L_1$ Lipschitz in the spectral norm. Using the fact that g is $\lambda$ -strongly convex, one has

\[g(y)-g(x)\ge \nabla g(x)^{\mathsf{T}}(y-x)+\frac{\lambda}{2}|y-x|^2.\]

Replacing $y=x_k$ and $x=x^*$ and using the fact that $\nabla g(x^*)=0$ , we have $g(x_k)-g(x^*)\ge \frac{\lambda}{2}|x_k-x^{*}|^2$ . Thus, from the final line, subtracting $g(x^*)$ to both sides, we get

\begin{align*} &\mathbb{E} \left(g(x_{n,k+1})-g(x^*)\right)\\ &\le \mathbb{E} \left(g(x_{n,k})-g(x^*)\right)-\lambda \gamma (2-L\gamma) \,\mathbb{E} \left(g(x_{n,k})-g(x^*)\right)\\ &\quad +\frac{2pLL^2_1\gamma^2}{m\lambda} \,\mathbb{E} \left(g(x_{n,k})-g(x^*)\right)+\frac{L\gamma^2}{m}\left|\left|\sigma(x^*)\right|\right|^2_F\\ &=\left[1-\lambda\gamma(2-L\gamma)+\frac{2pLL^2_1\gamma^2}{m\lambda}\right] \mathbb{E} \left(g(x_{n,k})-g(x^*)\right)+\frac{L\gamma^2}{m}\left|\left|\sigma(x^*)\right|\right|^2_F. \end{align*}

Note that due to our final assumption on $\gamma, \ m $ , $\left[1-\lambda\gamma(2-L\gamma)+\frac{2pLL^2_1\gamma^2}{m\lambda}\right]<1$ . Calling this quantity r, $\mathbb{E} \left(g(x_{n,k})-g(x^*)\right)=a_k$ and $\left|\left|\sigma(x^*)\right|\right|^2_F=B$ , we have the last line as

\begin{align*} a_{k+1}&\le r a_k+\frac{L\gamma^2}{m}B\\ & \le r^{k+1} a_0+\frac{L\gamma^2}{m}B\left(1+r+r^2+\cdots+r^k\right)\\ &\le r^{k+1} a_0+\frac{L\gamma^2}{m(1-r)}B. \end{align*}

Therefore, the first result for (5) follows from the last line. The second result for (5) follows using strong convexity of g, which implies that $g(y)-g(x^*)\ge \frac{\lambda}{2}|y-x^*|^2$ . The proof for (4) is exactly the same and hence we skip it.

Proof of Theorem 6. From the proof of Proposition 7, we know that

\begin{align*} \mathbb{E}\left|Y_{n,t}-D_{t}\right|^2&\le \underset{I*}{\underbrace{\left[4+4\left(t-k\gamma\right)\left(\frac{1}{m}\left(\mathbb{E}\left(h_1^2(u)\right)-L^2\right)+L^2\right)\right]\mathbb{E}\left|x_{n,k}-x_k\right|^2}}\\ &\quad+\underset{II*}{\underbrace{2\left[\frac{1}{m}4\left(t-k\gamma\right)^2+4\frac{\gamma}{m}\left(t-k\gamma\right)\right]\left(pL^2_1 \, \mathbb{E}\left|x_k-\tilde{x}_k\right|^2+\left|\left|\sigma(\tilde{x}_k)\right|\right|^2_F\right)}}. \end{align*}

Note that

\begin{align*} \mathbb{E}\left|x_{n,k}-x_k\right|^2&\le 2\left(\mathbb{E}\left|x_{n,k}-x^*\right|^2+\mathbb{E}\left|x^*-x_k\right|^2\right)\\ &\le \frac{8}{\lambda}\left[1-\lambda\gamma(2-L\gamma)+\frac{2pLL^2_1\gamma^2}{m\lambda}\right]^{k} \left(g(x_{0})-g(x^*)\right)\\ &\quad +\frac{8}{\lambda}\left[\frac{L\gamma}{m\left(\lambda(2-L\gamma)-\frac{2pLL^2_1\gamma}{m\lambda}\right)}\right] \left|\left|\sigma(x^*)\right|\right|^2_F. \end{align*}

In addition,

\begin{align*} \mathbb{E}\left|x_k-\tilde{x}_k\right|^2 &\le 2\left(\mathbb{E}\left|\tilde{x}_k-x^*\right|^2+\mathbb{E}\left|x^*-x_k\right|^2\right)\\ &\le \left[1-\gamma \lambda\left(2-L\gamma\right) \right]^k \left(g(x_0)-g(x^*)\right)\\&\quad+\frac{4}{\lambda}\left[1-\lambda\gamma(2-L\gamma)+\frac{2pLL^2_1\gamma^2}{m\lambda}\right]^{k} \left(g(x_{0})-g(x^*)\right)\\ &\quad +\frac{4}{\lambda}\left[\frac{L\gamma}{m\left(\lambda(2-L\gamma)-\frac{2pLL^2_1\gamma}{m\lambda}\right)}\right] \left|\left|\sigma(x^*)\right|\right|^2_F. \end{align*}

Hence, for the first term

\begin{align*} I^* &\le \frac{32}{\lambda}\,\left(1+\mathbb{E}\left(h_1^2(u)\right)\right)\, \left(g(x_0)-g(x^*)\right)\left[1-\lambda\gamma(2-L\gamma)+\frac{2pLL^2_1\gamma^2}{m\lambda}\right]^{k}\\ &\quad \quad +\frac{16}{\lambda^2\, m}\, L\, \left\|\sigma(x^*)\right\|^2_F\, \gamma \end{align*}

and for the second term

\begin{align*} II^* &\le 8\, p\, L^2_1\left[\left(1+\frac{4}{\lambda}\right)\left(g(x_0)-g(x^*)\right)+\frac{4\,L}{\lambda}\left\|\sigma(x^*)\right\|^2_F\right]\gamma^2\\ &\quad \quad +16\left(g(x_0)-g(x^*)+\left\|\sigma(x^*)\right\|^2_F \right)\gamma^2. \end{align*}

Therefore, we have

\begin{align*} W^2_2(Y_{n,t},D_{n,t})\le C^{**}_1\rho^k+C^{**}_2\gamma^2+C^{**}_3\gamma, \end{align*}

where

\begin{align*} &\rho=\left[1-\lambda\gamma(2-L\gamma)+\frac{2pLL^2_1\gamma^2}{m\lambda}\right]\!,\\ &C^{**}_1=\frac{32}{\lambda}\,\left(1+\mathbb{E}\left(h_1^2(u)\right)\right)\, \left(g(x_0)-g(x^*)\right)\!, \\ &C^{**}_2= 8\, p\, L^2_1\left[\left(1+\frac{4}{\lambda}\right)\left(g(x_0)-g(x^*)\right)+\frac{4\,L}{\lambda}\left\|\sigma(x^*)\right\|^2_F\right]\!,\\ &\quad \quad +16\left(g(x_0)-g(x^*)+\left\|\sigma(x^*)\right\|^2_F \right)\!,\\ &C^{**}_3=\frac{16}{\lambda^2\, m}\, L\, \left\|\sigma(x^*)\right\|^2_F. \end{align*}

Using the fact that

\begin{align*} W_2^2(Y_{n,t},X_t)& \le 2W_2^2(Y_{n,t},D_{t})+2W_2^2(D_{t},X_t) \end{align*}

we have

\begin{align*} W_2^2(Y_{n,t},X_t) &\le C^{**}_1\rho^k+C^{**}_2\gamma^2+C^{**}_3\gamma+2C_{11}\gamma^2+2C_{12}\frac{\gamma}{m}\\ &\le \tilde{C}^{**}_1\rho^k+\tilde{C}^{**}_2\gamma^2+\tilde{C}^{**}_3\gamma, \end{align*}

where

\begin{align*} &\tilde{C}^{**}_1=C^{**}_1,\\ & \tilde{C}^{**}_2=C^{**}_2+2\,C_{11},\\ & \tilde{C}^{**}_3=C^{**}_3+2\, C_{12}. \end{align*}