2.1. Approximation Problem

Consider independent normal, zero mean, random vectors X and $\epsilon$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon $$\end{document} , of respective dimensions k and n, where $k<n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k<n$$\end{document} , and with $C\text{ov}\left(X\right)=I_k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {C}}\hbox {ov}(X)=I_k$$\end{document} and $C\text{ov}\left(\epsilon \right)=D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {C}}\hbox {ov}(\varepsilon )=D$$\end{document} , a diagonal matrix. For any deterministic conformable matrix H, the n dimensional vector Y given by

(2.1) $\begin{array}{c}Y=HX+\epsilon \end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y=HX + \varepsilon \end{aligned}$$\end{document}

is called a standard FA model. The matrices (H, D) are the parameters that identify the model. As a consequence of the hypotheses,

(2.2) $\begin{array}{c}C\text{ov}\left(Y\right)=H{H}^{\top }+D.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathbb {C}}\hbox {ov}(Y) = HH^\top +D. \end{aligned}$$\end{document}

Given an n-dimensional covariance matrix $\hat{\Sigma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Sigma }$$\end{document} , one can pose the problem of approximating it with the covariance of a standard FA model, i.e., of finding (H, D) such that

(2.3) $\begin{array}{c}\hat{\Sigma }\approx H{H}^{\top }+D.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widehat{\Sigma } \approx HH^\top +D. \end{aligned}$$\end{document}

In this paper, we pose and solve the problem of finding an optimal approximation (2.3) when the criterion of closeness is the I-divergence (also known as Kullback-Leibler distance) between normal laws. Recall that [see e.g., Theorem 1.8.2 in Ihara (Reference Ihara1993)] if ${\nu }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nu _1$$\end{document} and ${\nu }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\nu _2$$\end{document} are two zero mean normal distributions on $R^m$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {R}}^m$$\end{document} , whose covariance matrices, ${\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _1$$\end{document} and ${\Sigma }_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _2$$\end{document} , respectively, are both non-singular, the I-divergence $I\left({\nu }_1\right|\left|{\nu }_2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ {\mathcal {I}}(\nu _1||\nu _2) $$\end{document} takes the explicit form ( $|·|$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$|\cdot |$$\end{document} denotes determinant)

(2.4) $\begin{array}{c}I({\nu }_1\left|\right|{\nu }_2)=\frac{1}{2}log\frac{|{\Sigma }_2|}{|{\Sigma }_1|}-\frac{m}{2}+\frac{1}{2}\mathrm{tr}\left({\Sigma }_2^{-1}{\Sigma }_1\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\nu _1||\nu _2)=\frac{1}{2}\log \frac{|\Sigma _2|}{|\Sigma _1|}-\frac{m}{2} +\frac{1}{2}\mathrm{tr}(\Sigma _2^{-1}\Sigma _1). \end{aligned}$$\end{document}

Since, because of zero means, the I-divergence depends only on the covariance matrices, we usually write $I\left({\Sigma }_1\right|\left|{\Sigma }_2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\Sigma _1||\Sigma _2)$$\end{document} instead of $I\left({\nu }_1\right|\left|{\nu }_2\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\nu _1||\nu _2)$$\end{document} . The approximate FA model problem can then be cast as follows.

Problem 2.1

Given the covariance matrix $\hat{\Sigma }>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Sigma }>0$$\end{document} , of size n, and an integer $k<n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k<n$$\end{document} , minimizeFootnote ¹

(2.5) $\begin{array}{c}I(\hat{\Sigma }\left|\right|H{H}^{\top }+D)=\frac{1}{2}log\frac{|H{H}^{\top }+D|}{|\hat{\Sigma }|}-\frac{n}{2}+\frac{1}{2}\mathrm{tr}\left({(H{H}^{\top }+D)}^{-1}\hat{\Sigma }\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\widehat{\Sigma } || HH^\top +D) = \frac{1}{2}\log \frac{|HH^\top +D|}{|\widehat{\Sigma }|}-\frac{n}{2} +\frac{1}{2}\mathrm{tr}((HH^\top +D)^{-1}\widehat{\Sigma }), \end{aligned}$$\end{document}

where the minimization is taken over all diagonal matrices $D\ge 0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D\ge 0$$\end{document} , and $H\in R^{n\times k}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H \in {\mathbb {R}}^{n\times k}$$\end{document} .

The first result is that in Problem 2.1, a minimum always exists.

2.2. Estimation Problem

In a statistical setup, the approximation Problem 2.1 has an equivalent formulation as an estimation problem. Let indeed $Y_1,\dots ,Y_N$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_1,\ldots ,Y_N$$\end{document} be a sequence of i.i.d. observations, whose distribution is modeled according to (2.1), where the matrices H and D are the unknown parameters. This is the context of Exploratory Factor Analysis, where no constraints are imposed on the matrix H. Let $\hat{\Sigma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Sigma }$$\end{document} denote the sample covariance matrix of the data. If the data have strictly positive covariance, for large enough N, the sample covariance will be strictly positive almost surely. The normal log likelihood $\ell (H,D)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell (H,D)$$\end{document} yields

(2.6) $\begin{array}{c}\frac{1}{N}\ell (H,D)=-\frac{n}{2}log\left(2\pi \right)-\frac{1}{2}log|H{H}^{\top }+D|-\frac{1}{2}\text{tr}({(H{H}^{\top }+D)}^{-1}\hat{\Sigma }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \frac{1}{N}\ell (H,D)=-\frac{n}{2}\log (2\pi )-\frac{1}{2}\log |HH^\top +D|-\frac{1}{2}\hbox {tr}\Big ((HH^\top + D)^{-1}\widehat{\Sigma }\Big ). \end{aligned}$$\end{document}

One immediately sees that $\ell (H,D)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\ell (H,D)$$\end{document} is, up to constants not depending on H and D, equal to $-I\left(\hat{\Sigma }\right||H{H}^{\top }+D)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-{\mathcal {I}}(\widehat{\Sigma }||HH^\top + D)$$\end{document} . Hence, maximum-likelihood estimation parallels I-divergence minimization in Problem 2.1, only the interpretation is different.

The equations for the maximum-likelihood estimators can be found in e.g., Section 14.3.1 of Anderson (Reference Anderson1984). In terms of the unknown parameters H and D, with D assumed to be non-singular, they are

(2.7) $\begin{array}{cc}H& =(\hat{\Sigma }-H{H}^{\top })D^{-1}H\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H&=(\widehat{\Sigma }-HH^\top )D^{-1}H \end{aligned}$$\end{document}

(2.8) $\begin{array}{cc}D& =\Delta (\hat{\Sigma }-H{H}^{\top }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D&=\Delta (\widehat{\Sigma }-HH^\top ). \end{aligned}$$\end{document}

where $\Delta \left(M\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Delta (M)$$\end{document} , defined for any square M, coincides with M on the diagonal and is zero elsewhere. Note that the matrix $H{H}^{\top }+D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$HH^\top +D$$\end{document} obtained by maximum-likelihood estimation is automatically invertible. Then it can be verified that Equation (2.7) is equivalent to

(2.9) $\begin{array}{c}H=\hat{\Sigma }{(H{H}^{\top }+D)}^{-1}H,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H =\widehat{\Sigma }(HH^\top +D)^{-1}H, \end{aligned}$$\end{document}

which is meaningful also when D is not invertible.

It is clear that the system of Equations (2.7), (2.8) does not have an explicit solution. For this reason, several numerical algorithms have been devised, among others a version of the EM algorithm, see Rubin and Thayer (Reference Rubin and Thayer1982). In the present paper we consider an alternative approach, which we will compare with the EM and some of the other algorithms.

Remark 3.1

The embedding (3.1) has a simple interpretation as a convenient reparametrization of the following alternative version of the standard FA model (2.1),

(3.3) $\begin{array}{c}Y=LZ+\epsilon ,\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y=LZ +\varepsilon , \end{aligned}$$\end{document}

where Z and $\epsilon$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\varepsilon $$\end{document} are zero mean normal vectors of sizes k and n, respectively, with $C\text{ov}\left(Z\right)=P>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {C}}\hbox {ov}(Z)=P>0$$\end{document} and $C\text{ov}\left(\epsilon \right)=I_n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathbb {C}}\hbox {ov}(\varepsilon )=I_n$$\end{document} . Letting $P=Q^{\top }Q$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P=Q^\top Q$$\end{document} and $X=Q^{-\top }Z$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X=Q^{-\top }Z$$\end{document} , where Q is any $k\times k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k\times k$$\end{document} square rootFootnote ² of P, one easily recognizes that (3.1) is a reparametrization of (3.3).

In this paper, all vectors are zero mean and normal, with law completely specified by the covariance matrix. The set of all covariance matrices of size $n+k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n+k$$\end{document} will be denoted as $\Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}$$\end{document} . An element $\Sigma \in \Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma \in {\varvec{\Sigma }}$$\end{document} can always be decomposed as

(3.4) $\begin{array}{c}\Sigma =\left(\begin{array}{cc}{\Sigma }_{11}& {\Sigma }_{12}\\ {\Sigma }_{21}& {\Sigma }_{22}\end{array}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Sigma =\begin{pmatrix} \Sigma _{11} &{} \Sigma _{12} \\ \Sigma _{21} &{} \Sigma _{22} \end{pmatrix}, \end{aligned}$$\end{document}

where ${\Sigma }_{11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{11}$$\end{document} and ${\Sigma }_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{22}$$\end{document} are square, of respective sizes n and k.

Two subsets of $\Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}$$\end{document} , comprising covariance matrices with special structure, will play a major role in what follows. The subset ${\Sigma }_0\subset \Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_0 \subset {\varvec{\Sigma }}$$\end{document} contains all covariances (3.4) with ${\Sigma }_{11}=\hat{\Sigma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{11}=\widehat{\Sigma }$$\end{document} , a given matrix, i.e.,

$\begin{array}{c}{\Sigma }_0=\{\Sigma \in \Sigma :{\Sigma }_{11}=\hat{\Sigma }\}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\varvec{\Sigma }}_0=\{\Sigma \in {\varvec{\Sigma }}: \Sigma _{11}=\widehat{\Sigma }\}. \end{aligned}$$\end{document}

The generic element of ${\Sigma }_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_0$$\end{document} will often be denoted as ${\Sigma }_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _0$$\end{document} . Also of interest is the subset ${\Sigma }_1\subset \Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_1\subset {\varvec{\Sigma }}$$\end{document} , containing all covariances (3.4) of the special form (3.2), i.e.,

$\begin{array}{c}{\Sigma }_1=\left(,\Sigma ,\in ,\Sigma ,:,,{\Sigma }_{11},=,H,H^{\top },+,D,,,,,{\Sigma }_{12},=,H,Q,,,,{\Sigma }_{22},=,Q^{\top },Q,,,\text{with},,,,H,,,D,,,Q,,\text{as in (3.2)},\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\varvec{\Sigma }}_1=\left\{ \, \Sigma \in {\varvec{\Sigma }}: \, \Sigma _{11}=HH^\top + D, \,\, \Sigma _{12}=HQ, \, \Sigma _{22}=Q^\top Q \, \hbox { with } \, \, H, D, Q \hbox { as in (3.2) }\right\} . \end{aligned}$$\end{document}

The generic elements of ${\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_1$$\end{document} will often be denoted ${\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _1$$\end{document} , or $\Sigma (H,D,Q)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma (H,D,Q)$$\end{document} when the parameters are relevant.

We are now ready to pose the following double minimization problem.

Problem 3.2

Find

$\begin{array}{c}\underset{{\Sigma }_0\in {\Sigma }_0,{\Sigma }_1\in {\Sigma }_1}{min}I({\Sigma }_0\left|\right|{\Sigma }_1).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{\Sigma _0\in {\varvec{\Sigma }}_0,\Sigma _1\in {\varvec{\Sigma }}_1}{\mathcal {I}}(\Sigma _0||\Sigma _1). \end{aligned}$$\end{document}

Problems 3.2 and 2.1 are related by the following proposition.

Proposition 3.3

Let $\hat{\Sigma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Sigma }$$\end{document} be given. It holds that

$\begin{array}{c}\underset{H,D}{min}I\left(\hat{\Sigma },,\left|\right|,,H,H^{\top },+,D\right)=\underset{{\Sigma }_0\in {\Sigma }_0,{\Sigma }_1\in {\Sigma }_1}{min}I\left({\Sigma }_0,\left|\right|,{\Sigma }_1\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{H,D}\, {\mathcal {I}}\left( \widehat{\Sigma }\,||\,HH^\top + D\right) =\min _{\Sigma _0\in {\varvec{\Sigma }}_0,\Sigma _1\in {\varvec{\Sigma }}_1}{\mathcal {I}}\left( \Sigma _0||\Sigma _1\right) . \end{aligned}$$\end{document}

Proof

The proof can be found in Finesso and Spreij (Reference Finesso, Chiuso, Pinzoni and Ferrante2007). $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

3.1. Partial Minimization Problems

The first partial minimization, required for the solution of Problem 3.2, is as follows.

Problem 3.4

Given a strictly positive definite covariance matrix $\Sigma \in \Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma \in {\varvec{\Sigma }}$$\end{document} , find

$\begin{array}{c}\underset{{\Sigma }_0\in {\Sigma }_0}{min}I({\Sigma }_0\left|\right|\Sigma ).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{\Sigma _0 \in {\varvec{\Sigma }}_0} \, {\mathcal {I}}(\Sigma _0||\Sigma ). \end{aligned}$$\end{document}

The unique solution to this problem can be computed analytically and is given below.

Proposition 3.5

The unique minimizer ${\Sigma }_0^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _0^*$$\end{document} of Problem 3.4 is given by

$\begin{array}{c}{\Sigma }_0^{\ast }=\left(\begin{array}{cc}\hat{\Sigma }& \hat{\Sigma }{\Sigma }_{11}^{-1}{\Sigma }_{12}\\ {\Sigma }_{21}{\Sigma }_{11}^{-1}\hat{\Sigma }& {\Sigma }_{22}-{\Sigma }_{21}{\Sigma }_{11}^{-1}({\Sigma }_{11}-\hat{\Sigma }){\Sigma }_{11}^{-1}{\Sigma }_{12}\end{array}\right)>0.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Sigma _0^*=\begin{pmatrix} \widehat{\Sigma } &{} \,\, \widehat{\Sigma }\Sigma _{11}^{-1}\Sigma _{12}\\ \Sigma _{21}\Sigma _{11}^{-1}\widehat{\Sigma } &{} \,\, \Sigma _{22}-\Sigma _{21}\Sigma _{11}^{-1} (\Sigma _{11}-\widehat{\Sigma })\Sigma _{11}^{-1}\Sigma _{12} \end{pmatrix} >0. \end{aligned}$$\end{document}

Moreover,

(3.5) $\begin{array}{c}I({\Sigma }_0^{\ast }\left|\right|\Sigma )=I(\hat{\Sigma }\left|\right|{\Sigma }_{11}),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\Sigma _0^*||\Sigma )={\mathcal {I}}(\widehat{\Sigma }||\Sigma _{11}), \end{aligned}$$\end{document}

and the Pythagorean rule

$\begin{array}{c}I({\Sigma }_0\left|\right|\Sigma )=I({\Sigma }_0\left|\right|{\Sigma }_0^{\ast })+I({\Sigma }_0^{\ast }\left|\right|\Sigma )\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\Sigma _0||\Sigma )={\mathcal {I}}(\Sigma _0||\Sigma _0^*)+{\mathcal {I}}(\Sigma _0^*||\Sigma ) \end{aligned}$$\end{document}

holds for any strictly positive ${\Sigma }_0\in {\Sigma }_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _0\in {\varvec{\Sigma }}_0$$\end{document} .

Problem 3.6

Given a strictly positive definite covariance matrix $\Sigma \in \Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma \in {\varvec{\Sigma }}$$\end{document} , find

$\begin{array}{c}\underset{{\Sigma }_1\in {\Sigma }_1}{min}I\left(\Sigma \right||{\Sigma }_1).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{\Sigma _1 \in {\varvec{\Sigma }}_1} \, {\mathcal {I}}(\Sigma ||\Sigma _1). \end{aligned}$$\end{document}

The proposition below gives the explicit solution to this problem.

Proposition 3.7

A minimizer ${\Sigma }_1^{\ast }=\Sigma (H^{\ast },D^{\ast },Q^{\ast })$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _1^*=\Sigma (H^*,D^*,Q^*)$$\end{document} of Problem 3.6 is given by

$\begin{array}{cc}{Q}^{\ast }& ={\Sigma }_{22}^{1/2}\\ H^{\ast }& ={\Sigma }_{12}{\Sigma }_{22}^{-1/2}\\ D^{\ast }& =\Delta \left({\tilde{\Sigma }}_{11}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Q^*&=\Sigma _{22}^{1/2}\\ H^*&= \Sigma _{12}\Sigma _{22}^{-1/2} \\ D^*&= \Delta (\widetilde{\Sigma }_{11}), \end{aligned}$$\end{document}

where

$\begin{array}{c}{\tilde{\Sigma }}_{11}={\Sigma }_{11}-{\Sigma }_{12}{\Sigma }_{22}^{-1}{\Sigma }_{21}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \widetilde{\Sigma }_{11}=\Sigma _{11}-\Sigma _{12}\Sigma _{22}^{-1}\Sigma _{21}. \end{aligned}$$\end{document}

The corresponding minimizing matrix is

(3.6) $\begin{array}{c}{\Sigma }_1^{\ast }=\Sigma (H^{\ast },D^{\ast },Q^{\ast })=\left(\begin{array}{cc}{\Sigma }_{12}{\Sigma }_{22}^{-1}{\Sigma }_{21}+\Delta \left({\tilde{\Sigma }}_{11}\right)& {\Sigma }_{12}\\ {\Sigma }_{21}& {\Sigma }_{22}\end{array}\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Sigma _1^*=\Sigma (H^*,D^*,Q^*)= \begin{pmatrix} \Sigma _{12}\Sigma _{22}^{-1}\Sigma _{21}+\Delta (\widetilde{\Sigma }_{11}) &{} \Sigma _{12} \\ \Sigma _{21} &{} \Sigma _{22} \end{pmatrix}. \end{aligned}$$\end{document}

Moreover, $I\left(\Sigma \right||{\Sigma }_1^{\ast })=I({\tilde{\Sigma }}_{11}\left|\right|\Delta \left({\tilde{\Sigma }}_{11}\right))$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\Sigma ||\Sigma _1^*)= {\mathcal {I}}(\widetilde{\Sigma }_{11}||\Delta (\widetilde{\Sigma }_{11}))$$\end{document} and the Pythagorean rule

(3.7) $\begin{array}{c}I\left(\Sigma \right||{\Sigma }_1)=I(\Sigma \left|\right|{\Sigma }_1^{\ast })+I({\Sigma }_1^{\ast }\left|\right|{\Sigma }_1)\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\Sigma ||\Sigma _1)={\mathcal {I}}(\Sigma ||\Sigma _1^*)+{\mathcal {I}}(\Sigma _1^*||\Sigma _1) \end{aligned}$$\end{document}

holds for any ${\Sigma }_1=\Sigma (H,D,Q)\in {\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _1=\Sigma (H,D,Q)\in {\varvec{\Sigma }}_1$$\end{document} .

Remark 3.8

Note that, since $\Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma $$\end{document} is supposed to be strictly positive, ${\tilde{\Sigma }}_{11}:={\Sigma }_{11}-{\Sigma }_{12}{\Sigma }_{22}^{-1}{\Sigma }_{21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{\Sigma }_{11}:=\Sigma _{11}-\Sigma _{12}\Sigma _{22}^{-1}\Sigma _{21}$$\end{document} is strictly positive too. It follows that $D^{\ast }=\Delta \left({\tilde{\Sigma }}_{11}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D^*=\Delta (\widetilde{\Sigma }_{11})$$\end{document} is strictly positive.

We close this section by considering a constrained version of the second partial minimization Problem 3.6 to which we will return in Section 5, when we discuss the connection with the EM algorithm. The constraint that we impose is $Q=Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q=Q_0$$\end{document} fixed, whereas H and D remain free. The set over which the optimization will be carried out is ${\Sigma }_{10}\subset {\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{10}\subset {\varvec{\Sigma }}_1$$\end{document} defined as

$\begin{array}{c}{\Sigma }_{10}=\left(,\Sigma ,\in ,\Sigma ,,:,,{\Sigma }_{11},=,H,H^{\top },+,D,,,,,{\Sigma }_{12},=,H,Q_0,,,,{\Sigma }_{22},=,Q_0^{\top },Q_0,,,\text{with},,,,H,,,D,,\text{as in},,,\left(3.2\right),\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\varvec{\Sigma }}_{10}=\left\{ \, \Sigma \in {\varvec{\Sigma }}\, : \, \Sigma _{11}=HH^\top + D, \,\, \Sigma _{12}=HQ_0, \, \Sigma _{22}=Q_0^\top Q_0 \, \hbox { with } \, \, H, D \hbox { as in } \,(3.2)\, \right\} . \end{aligned}$$\end{document}

We pose the following constrained optimization problem.

Problem 3.9

Given a strictly positive covariance $\Sigma \in \Sigma$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma \in {\varvec{\Sigma }}$$\end{document} , find

$\begin{array}{c}\underset{{\Sigma }_{10}\in {\Sigma }_{10}}{min}I\left(\Sigma \right||{\Sigma }_{10}).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \min _{\Sigma _{10} \in {\varvec{\Sigma }}_{10}} \, {\mathcal {I}}(\Sigma ||\Sigma _{10}). \end{aligned}$$\end{document}

The solution is given in the next proposition.

Proposition 3.10

A solution ${\Sigma }_{10}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{10}^*$$\end{document} of Problem 3.9 is obtained for $H^{\ast }={\Sigma }_{12}{\Sigma }_{22}^{-1}{Q}_0^{\top }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H^*=\Sigma _{12}\Sigma _{22}^{-1}Q_0^\top $$\end{document} , and $D^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D^*$$\end{document} as in Proposition 3.7, resulting in

(3.8) $\begin{array}{c}{\Sigma }_{10}^{\ast }=\left(\begin{array}{cc}{\Sigma }_{12}{\Sigma }_{22}^{-1}{Q}_0^{\top }{Q}_0{\Sigma }_{22}^{-1}{\Sigma }_{21}+\Delta \left({\tilde{\Sigma }}_{11}\right)& {\Sigma }_{12}{\Sigma }_{22}^{-1}{Q}_0^{\top }{Q}_0\\ Q_0^{\top }{Q}_0{\Sigma }_{22}^{-1}{\Sigma }_{21}& Q_0^{\top }{Q}_0\end{array}\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \Sigma _{10}^*=\begin{pmatrix} \Sigma _{12}\Sigma _{22}^{-1}Q_0^\top Q_0\Sigma _{22}^{-1}\Sigma _{21} + \Delta (\widetilde{\Sigma }_{11}) &{} \Sigma _{12}\Sigma _{22}^{-1}Q_0^\top Q_0 \\ Q_0^\top Q_0\Sigma _{22}^{-1}\Sigma _{21} &{} Q_0^\top Q_0 \end{pmatrix}. \end{aligned}$$\end{document}

Proof

See Appendix 2. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

For the constrained problem, the Pythagorean rule does not hold. Intuitively, since ${\Sigma }_{10}\subset {\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{10}\subset {\varvec{\Sigma }}_1$$\end{document} the optimal value of the constrained Problem 3.9 is in general higher than the optimal value of the free Problem 3.6. To compute the extra cost incurred notice that ${\Sigma }_{10}^{\ast }\in {\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{10}^* \in {\varvec{\Sigma }}_1$$\end{document} , therefore the Pythagorean rule (3.7) gives

(3.9) $\begin{array}{c}I\left(\Sigma \right||{\Sigma }_{10}^{\ast })=I(\Sigma \left|\right|{\Sigma }_1^{\ast })+I({\Sigma }_1^{\ast }\left|\right|{\Sigma }_{10}^{\ast }),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\Sigma ||\Sigma _{10}^*)= {\mathcal {I}}(\Sigma ||\Sigma _1^*)+{\mathcal {I}}(\Sigma _1^*||\Sigma _{10}^*), \end{aligned}$$\end{document}

hence $I\left(\Sigma \right||{\Sigma }_{10}^{\ast })\ge I(\Sigma \left|\right|{\Sigma }_1^{\ast })$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\Sigma ||\Sigma _{10}^*)\ge {\mathcal {I}}(\Sigma ||\Sigma _1^*)$$\end{document} , where ${\Sigma }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _1^*$$\end{document} is as in Proposition 3.7. The quantity $I\left({\Sigma }_1^{\ast }\right|\left|{\Sigma }_{10}^{\ast }\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\Sigma _1^*||\Sigma _{10}^*)$$\end{document} represents the extra cost. An elementary computation gives

(3.10) $\begin{array}{c}I({\Sigma }_1^{\ast }\left|\right|{\Sigma }_{10}^{\ast })=I({\Sigma }_{22}\left|\right|Q_0^{\top }{Q}_0),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\Sigma _1^*||\Sigma _{10}^*)={\mathcal {I}}(\Sigma _{22}||Q_0^\top Q_0), \end{aligned}$$\end{document}

i.e., the optimizing matrices ${\Sigma }_1^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _1^*$$\end{document} and ${\Sigma }_{10}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{10}^*$$\end{document} , see (3.6), (3.8), coincide iff $Q_0^{\top }{Q}_0={\Sigma }_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0^\top Q_0=\Sigma _{22}$$\end{document} .

Summarizing the comments on the constrained problem: (i) the optimal value at the minimum is higher since ${\Sigma }_{10}\subset {\Sigma }_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\varvec{\Sigma }}_{10}\subset {\varvec{\Sigma }}_1$$\end{document} , (ii) the extra cost is explicitly given by (3.10), as $I\left({\Sigma }_{22}\right|\left|Q_0^{\top }{Q}_0\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\Sigma _{22}||Q_0^\top Q_0)$$\end{document} , and (iii) there is no analog to the Pythagorean rule (3.7). The conclusion is that the solution ${\Sigma }_{10}^{\ast }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{10}^*$$\end{document} of the constrained Problem 3.9 is suboptimal for the free Problem 3.6. The consequences of the suboptimality will be further discussed in Section 5.

4.1. The Algorithm

We suppose that the given covariance matrix $\hat{\Sigma }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Sigma }$$\end{document} is strictly positive definite. To set up the iterative minimization algorithm, assign initial values $H_0,D_0,Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_0, D_0, Q_0$$\end{document} to the parameters, with $D_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_0$$\end{document} diagonal, $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} invertible and $H_0{H}_0^{\top }+D_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_0H_0^\top +D_0$$\end{document} invertible. The updating rules are constructed as follows. Let $H_t,D_t,Q_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_t, D_t, Q_t$$\end{document} be the parameters at the t-th iteration, and ${\Sigma }_{1,t}=\Sigma (H_t,D_t,Q_t)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{1,t} = \Sigma (H_t,D_t,Q_t)$$\end{document} the corresponding covariance, defined as in (3.2). Now solve the two partial minimizations as illustrated below.

where ${\Sigma }_{0,t}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{0,t}$$\end{document} denotes the solution of the first minimization with input ${\Sigma }_{1,t}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Sigma _{1,t}$$\end{document} . To express in a compact form the resulting update equations, define

(4.1) $\begin{array}{c}{R}_t=I-H_t^{\top }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t+H_t^{\top }{(H_t{H}_t^{\top }+D_t)}^{-1}\hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_t=I- H_t^\top (H_tH_t^\top + D_t)^{-1}H_t + H_t^\top (H_tH_t^\top + D_t)^{-1} \widehat{\Sigma } (H_tH_t^\top + D_t)^{-1}H_t. \end{aligned}$$\end{document}

Note that, by Remark 3.8, $H_t{H}_t^{\top }+D_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_tH^\top _t+D_t$$\end{document} is actually invertible for all t, since both $H_0{H}_0^{\top }+D_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_0H^\top _0+D_0$$\end{document} and $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} have been chosen to be invertible. It is easy to show that also $I-H_t^{\top }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$I- H_t^\top (H_tH_t^\top + D_t)^{-1}H_t$$\end{document} , and consequently $R_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} , are strictly positive and therefore invertible. The update equations resulting from the cascade of the two minimizations are

(4.2) $\begin{array}{cc}{Q}_{t+1}& =(Q_t^{\top }{R}_t{Q}_t{)}^{1/2},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Q_{t+1}&= \Big (Q_t^\top R_t Q_t\Big )^{1/2}, \end{aligned}$$\end{document}

(4.3) $\begin{array}{cc}{H}_{t+1}& =\hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t{Q}_t{Q}_{t+1}^{-1},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H_{t+1}&= \widehat{\Sigma }(H_tH_t^\top + D_t)^{-1}H_tQ_tQ_{t+1}^{-1}, \end{aligned}$$\end{document}

(4.4) $\begin{array}{cc}{D}_{t+1}& =\Delta (\hat{\Sigma }-H_{t+1}{H}_{t+1}^{\top }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{t+1}&= \Delta (\widehat{\Sigma }-H_{t+1}H_{t+1}^\top ). \end{aligned}$$\end{document}

Properly choosing the square root in Equation (4.2) will make $Q_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_t$$\end{document} disappear from the update equations. This is an attractive feature since, at the t-th step of the algorithm, only $H_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_t$$\end{document} and $D_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_t$$\end{document} are needed to construct the approximation $H_t{H}_t^{\top }+D_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_t H_t^\top + D_t$$\end{document} . The choice that accomplishes this is ${\left(Q_t^{\top }{R}_t{Q}_t\right)}^{1/2}=R_t^{1/2}{Q}_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(Q_t^\top R_t Q_t)^{1/2} = R_t^{1/2} Q_t$$\end{document} , where $R_t^{1/2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t^{1/2}$$\end{document} is a symmetric root of $R_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} , resulting in $Q_{t+1}=R_t^{1/2}{Q}_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_{t+1} = R_t^{1/2} Q_t$$\end{document} . Upon substituting $Q_{t+1}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_{t+1}$$\end{document} in Equation (4.3), one gets the AML algorithm.

Algorithm 4.1

(AML) Given $H_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_t$$\end{document} , $D_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_t$$\end{document} from the t-th step, and $R_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} as in (4.1), the update equations for a I-divergence minimizing algorithm are

(4.5) $\begin{array}{cc}{H}_{t+1}& =\hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t{R}_t^{-1/2}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H_{t+1}&= \widehat{\Sigma }(H_tH_t^\top + D_t)^{-1}H_tR_t^{-1/2} \end{aligned}$$\end{document}

(4.6) $\begin{array}{cc}{D}_{t+1}& =\Delta (\hat{\Sigma }-H_{t+1}{H}_{t+1}^{\top }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{t+1}&= \Delta (\widehat{\Sigma }-H_{t+1}H_{t+1}^\top ). \end{aligned}$$\end{document}

Since $R_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} only depends on $H_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_t$$\end{document} and $D_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_t$$\end{document} , see (4.1), the parameter $Q_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_t$$\end{document} has been effectively removed from the update equations, although its presence was essential for the derivation.

Proposition 4.3

Let $H_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_t$$\end{document} be as in Algorithm 4.1. Pick $H_0=H_0{H}_0^{\top }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {H}}_0=H_0H_0^\top $$\end{document} , and $D_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_0$$\end{document} such that $H_0+D_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {H}}_0+D_0$$\end{document} is invertible. The update equation for $H_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {H}}_{t}$$\end{document} becomes

(4.7) $\begin{array}{c}{H}_{t+1}=\hat{\Sigma }{(H_t+D_t)}^{-1}{H}_t(D_t+\hat{\Sigma }{(H_t+D_t)}^{-1}{H}_t{)}^{-1}\hat{\Sigma }.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {H}}_{t+1} = \widehat{\Sigma }({\mathcal {H}}_{t}+D_{t})^{-1}{\mathcal {H}}_{t}\big (D_{t} +\widehat{\Sigma }({\mathcal {H}}_t+D_t)^{-1}{\mathcal {H}}_t\big )^{-1}\widehat{\Sigma }. \end{aligned}$$\end{document}

4.2. Asymptotic Properties

In Proposition 4.4 below, we collect the asymptotic properties of Algorithm 4.1, also quantifying the I-divergence decrease at each step.

Algorithm 5.1

(EM)

$\begin{array}{cc}{H}_{t+1}& =\hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t{R}_t^{-1}\\ D_{t+1}& =\Delta (\hat{\Sigma }-H_{t+1}{R}_t{H}_{t+1}^{\top }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H_{t+1}&= \widehat{\Sigma }(H_t H_t^\top + D_t)^{-1}H_t R_t^{-1} \\ D_{t+1}&= \Delta (\widehat{\Sigma }- H_{t+1} R_t H_{t+1}^\top ). \end{aligned}$$\end{document}

The EM Algorithm 5.1 differs in both equations from our AML Algorithm 4.1. It is well known that EM algorithms can be derived as alternating minimizations, see Csiszár and Tusnády (Reference Csiszár and Tusnády1984), it is therefore interesting to investigate how Algorithm 5.1 can be derived within our framework. Thereto one considers the first partial minimization problem together with the constrained second partial minimization Problem 3.9, the constraint being $Q=Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q=Q_0$$\end{document} , for some $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} . Later on we will see that the particular choice of $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} , as long as it is invertible, is irrelevant. The concatenation of these two problems results in the EM Algorithm 5.1, as is detailed below.

Starting at $(H_t,D_t,Q_0)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(H_t,D_t,Q_0)$$\end{document} , one performs the first partial minimization that results in the matrix

$\begin{array}{c}\left(\begin{array}{cc}\hat{\Sigma }& \hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t{Q}_0\\ Q_0^{\top }{H}_t^{\top }{(H_t{H}_t^{\top }+D_t)}^{-1}\hat{\Sigma }& Q_0^{\top }{R}_t{Q}_0\end{array}\right).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{pmatrix} \widehat{\Sigma } &{} \widehat{\Sigma }(H_tH_t^\top +D_t)^{-1}H_tQ_0 \\ Q_0^\top H_t^\top (H_tH_t^\top +D_t)^{-1}\widehat{\Sigma } &{} Q_0^\top R_t Q_0 \end{pmatrix}. \end{aligned}$$\end{document}

Performing now the constrained second minimization, according to the results of Proposition 3.10, one obtains

(5.1) $\begin{array}{cc}{H}_{t+1}& =\hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t{R}_t^{-1}\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H_{t+1}&= \widehat{\Sigma }(H_tH_t^\top +D_t)^{-1}H_t R_t^{-1} \end{aligned}$$\end{document}

(5.2) $\begin{array}{cc}{D}_{t+1}& =\Delta (\hat{\Sigma }-\hat{\Sigma }{(H_t{H}_t^{\top }+D_t)}^{-1}{H}_t{R}_t^{-1}{H}_t^{\top }{(H_t{H}_t^{\top }+D_t)}^{-1}\hat{\Sigma }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{t+1}&= \Delta \big (\widehat{\Sigma }-\widehat{\Sigma }(H_tH_t^\top +D_t)^{-1}H_t R_t ^{-1} H_t^\top (H_tH_t^\top +D_t)^{-1}\widehat{\Sigma }\big ). \end{aligned}$$\end{document}

Substitution of (5.1) into (5.2) yields

(5.3) $\begin{array}{c}{D}_{t+1}=\Delta (\hat{\Sigma }-H_{t+1}{R}_t{H}_{t+1}^{\top }).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D_{t+1}=\Delta (\widehat{\Sigma }-H_{t+1}R_tH_{t+1}^\top ). \end{aligned}$$\end{document}

One sees that the matrix $Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_0$$\end{document} does not appear in the recursion, just as the matrices $Q_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q_t$$\end{document} do not occur in Algorithm 4.1, but we lost the second optimality property in the construction of Algorithm 4.1, due to the imposed constraint $Q=Q_0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Q=Q_0$$\end{document} . Moreover, the EM algorithm does not enjoy the diagonal preservation property mentioned in Remark 4.2 for Algorithm 4.1, due to the presence of $R_t$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_t$$\end{document} in Equation (5.3).

Summarizing, both Algorithms 4.1 and 5.1 are the result of two partial minimization problems. The latter algorithm differs from ours in that the second partial minimization is constrained. In view of the extra cost incurred by the suboptimal constrained minimization, see Equation (3.9), our Algorithm 4.1 yields a better performance. We will illustrate these considerations by some numerical examples in Section 7.

6.1. Approximation with Singular D

In this section, we consider the approximation Problem 2.1 under the constraint $D_2=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_2=0$$\end{document} where

(6.1) $\begin{array}{c}D=\left(\begin{array}{cc}{D}_1& 0\\ 0& D_2\end{array}\right)=\left(\begin{array}{cc}\tilde{D}& 0\\ 0& 0\end{array}\right),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} D=\begin{pmatrix} D_1 &{} 0 \\ 0 &{} D_2 \end{pmatrix}=\begin{pmatrix} \widetilde{D} &{} 0 \\ 0 &{} 0 \end{pmatrix}, \end{aligned}$$\end{document}

with $D_1=\tilde{D}>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_1=\widetilde{D}>0$$\end{document} of size $n_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_1$$\end{document} and $D_2=0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$D_2=0$$\end{document} of size $n_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2$$\end{document} . The constrained minimization problem can be formulated as follows.

Problem 6.1

Given $\hat{\Sigma }>0$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widehat{\Sigma }>0$$\end{document} of size $n\times n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n\times n$$\end{document} and integers $n_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2$$\end{document} and k, with $n_2\le k<n$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_2 \le k < n$$\end{document} , minimize

$\begin{array}{c}I\left(\hat{\Sigma }\right||H{H}^{\top }+D),\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\widehat{\Sigma } || HH^\top +D), \end{aligned}$$\end{document}

over (H, D) with D satisfying (6.1).

Remark 6.2

Alternatively, in Jöreskog (Reference Jöreskog1967), the solution of the likelihood Equations (2.8) and (2.9) has been investigated under zero constraints on D. In this section, we work directly on the objective function of Problem 6.1.

To reduce the complexity of Problem 6.1 we will now decompose the objective function, choosing a convenient representation of the matrix $H=\left(\begin{array}{c}{H}_1\\ H_2\end{array}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H=\begin{pmatrix} H_1 \\ H_2 \end{pmatrix}$$\end{document} , where $H_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_i$$\end{document} has $n_i$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_i$$\end{document} rows. Inspired by the parametrization in Jöreskog (Reference Jöreskog1967) we make the following observation. Given any orthogonal matrix Q, define $H^{\prime }=HQ$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H' = HQ$$\end{document} , then clearly $H^{\prime }{H}^{\prime \top }+D=H{H}^{\top }+D$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H'H'^\top + D = HH^\top +D$$\end{document} . Let $H_2=U\left(0\Lambda \right)V^{\top }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}=U(0\,\, \Lambda )V^{\top }$$\end{document} be the singular value decomposition of $H_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}$$\end{document} , with $\Lambda$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\Lambda $$\end{document} a positive definite diagonal matrix of size $n_2\times n_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{2}\times n_{2}$$\end{document} , and U and V orthogonal of sizes $n_2\times n_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{2}\times n_{2}$$\end{document} and $k\times k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$k\times k$$\end{document} , respectively. Let

$\begin{array}{c}{H}^{\prime }=HV\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H'= HV \end{aligned}$$\end{document}

The blocks of $H^{\prime }$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H'$$\end{document} are $H_1^{\prime }=H_{1}V$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{1}' =H_{1}V$$\end{document} and $H_2^{\prime }=\left(H_{21}^{\prime }{H}_{22}^{\prime }\right):=\left(0U\Lambda \right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}'=(H_{21}'\, H_{22}'):=(0\quad U\Lambda )$$\end{document} , with $H_{21}^{\prime }\in R^{(k-n_2)\times n_2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{21}'\in {\mathbb {R}}^{(k-n_2)\times n_2}$$\end{document} and $H_{22}^{\prime }\in R^{n_2\times n_2}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H'_{22}\in {\mathbb {R}}^{n_2\times n_2}$$\end{document} . Hence, without loss of generality, in the remainder of this section we assume that

(6.2) $\begin{array}{c}H=\left(\begin{array}{c}{H}_1\\ H_2\end{array}\right)=\left(\begin{array}{cc}{H}_{11}& H_{12}\\ 0& H_{22}\end{array}\right),H_{22}\text{invertible}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} H=\begin{pmatrix} H_1 \\ H_2 \end{pmatrix} = \begin{pmatrix} H_{11} &{} H_{12} \\ 0 &{} H_{22} \end{pmatrix}, \qquad H_{22} \,\, \hbox {invertible}. \end{aligned}$$\end{document}

Finally, let

$\begin{array}{c}K={\hat{\Sigma }}_{12}{\hat{\Sigma }}_{22}^{-1}-H_1{H}_2^{\top }{\left(H_2{H}_2^{\top }\right)}^{-1},\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} K=\widehat{\Sigma }_{12}\widehat{\Sigma }_{22}^{-1}-H_1H_2^\top (H_2H_2^\top )^{-1}, \end{aligned}$$\end{document}

which, under (6.2), is equivalent to

$\begin{array}{c}K={\hat{\Sigma }}_{12}{\hat{\Sigma }}_{22}^{-1}-H_{12}{H}_{22}^{-1}.\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} K=\widehat{\Sigma }_{12}\widehat{\Sigma }_{22}^{-1}-H_{12}H_{22}^{-1}. \end{aligned}$$\end{document}

Here is the announced decomposition of the objective function.

Lemma 6.3

Let D be as in Equation (6.1). The following I-divergence decomposition holds.

(6.3) $\begin{array}{cc}I\left(\hat{\Sigma }\right||H{H}^{\top }+D)=& I({\tilde{\Sigma }}_{11}\left|\right|H_{11}{H}_{11}^{\top }+\tilde{D})+I({\hat{\Sigma }}_{22}\left|\right|H_{22}{H}_{22}^{\top })\\ & +\frac{1}{2}\text{tr}({\hat{\Sigma }}_{22}{K}^{\top }{(H_{11}{H}_{11}^{\top }+\tilde{D})}^{-1}K).\end{array}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} {\mathcal {I}}(\widehat{\Sigma }||HH^{\top }+D) =&\, {\mathcal {I}}(\widetilde{\Sigma }_{11}||H_{11}H_{11}^{\top }+ \widetilde{D})+{\mathcal {I}}(\widehat{\Sigma }_{22}||H_{22}H_{22}^{\top })\nonumber \\&+\frac{1}{2}\text{ tr }\big (\widehat{\Sigma }_{22}K^\top (H_{11}H_{11}^{\top } +\widetilde{D})^{-1}K\big ). \end{aligned}$$\end{document}

Proof

The proof follows from Lemma 9.1. $\square$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\square $$\end{document}

We are now ready to characterize the solution of Problem 6.1. Observe first that the second and third terms on the right-hand side of (6.3) are non-negative and can be made zero. To this end, it is enough to select $H_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{22}$$\end{document} such that $H_{22}{H}_{22}^{\top }={\hat{\Sigma }}_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{22}H_{22}^{\top }=\widehat{\Sigma }_{22}$$\end{document} and then $H_{12}={\hat{\Sigma }}_{12}{\hat{\Sigma }}_{22}^{-1}{H}_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{12}=\widehat{\Sigma }_{12}\widehat{\Sigma }_{22}^{-1}H_{22}$$\end{document} . The remaining blocks, $H_{11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{11}$$\end{document} and $\tilde{D}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{D}$$\end{document} , are determined minimizing the first term. We have thus proved the following

Proposition 6.4

Any pair (H, D), as in (6.1) and (6.2), solving Problem 6.1 satisfies

1. (i) $I\left({\tilde{\Sigma }}_{11}\right||H_{11}{H}_{11}^{\top }+\tilde{D})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\widetilde{\Sigma }_{11}||H_{11}H_{11}^{\top }+\widetilde{D})$$\end{document} is minimized,

2. (ii) $I(\hat{\Sigma }\left|\right|H{H}^{\top }+D)=I({\tilde{\Sigma }}_{11}\left|\right|H_{11}{H}_{11}^{\top }+\tilde{D})$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\widehat{\Sigma }||HH^\top + D) = {\mathcal {I}}(\widetilde{\Sigma }_{11}||H_{11}H_{11}^{\top }+\widetilde{D})$$\end{document} ,

3. (iii) $H_{22}{H}_{22}^{\top }={\hat{\Sigma }}_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{22}H_{22}^{\top } = \widehat{\Sigma }_{22}$$\end{document} ,

4. (iv) $H_{12}={\hat{\Sigma }}_{12}{\hat{\Sigma }}_{22}^{-1}{H}_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{12} = \widehat{\Sigma }_{12}\widehat{\Sigma }_{22}^{-1}H_{22}$$\end{document} .

Remark 6.5

In the special case $n_2=k$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_{2}=k$$\end{document} , the matrices $H_{11}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{11}$$\end{document} and $H_{21}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{21}$$\end{document} are empty, $H_{12}=H_1$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{12}=H_{1}$$\end{document} , and $H_{22}=H_2$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{22}=H_{2}$$\end{document} . From Proposition 6.4, at the minimum, $H_2{H}_2^{\top }={\hat{\Sigma }}_{22}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{2}H_{2}^{\top }=\widehat{\Sigma }_{22}$$\end{document} , $H_1{H}_2^{\top }={\hat{\Sigma }}_{12}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$H_{1}H_2^\top =\widehat{\Sigma }_{12}$$\end{document} , and $\tilde{D}$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{D}$$\end{document} minimizes $I\left({\tilde{\Sigma }}_{11}\right|\left|\tilde{D}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${\mathcal {I}}(\widetilde{\Sigma }_{11}||\widetilde{D})$$\end{document} . The latter problem has solution $\tilde{D}=\Delta \left({\tilde{\Sigma }}_{11}\right)$ \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\widetilde{D}=\Delta (\widetilde{\Sigma }_{11})$$\end{document} . It is remarkable that in this case the minimization problem has an explicit solution.