2 Proof of Theorem 1

Let $\mathbb {G}_{n,k}$ be the Grassmannian of k-dimensional subspaces of $\mathbb {R}^n$ . Given $g\in \mathbb {G}_{n,k}$ , let $\operatorname {\mathrm {O}}(g)$ be the subgroup of ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ that fixes g. For $A\in \operatorname {\mathrm {GL}}_n(\mathbb {R})$ , we denote by $A_\#$ the mapping corresponding to the natural induced action on $\mathbb {G}_{n,k}$ and by $A|_g$ the restriction of A to the subspace g. Choose orthonormal bases for g and the image of g under A and let $\det A|_g$ denote the determinant of the matrix representing A with respect to these bases. It is easy to see that the absolute value $|{\det}\ A|_g|$ is independent of the choice of bases.

Consider the Riemannian metric on ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ coming from its embedding in the space of $n\times n$ matrices with the natural inner product $\langle A,B\rangle =\operatorname {\mathrm {tr}}(A\,{\vphantom {B}}^{t}\!{B}).$ As a Lie group, this Riemannian structure on ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ is left and right invariant and it induces a Riemannian structure on $\mathbb {G}_{n,k}$ as an homogeneous space of ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ . We denote by $\mathrm {vol} \operatorname {\mathrm {O}}_n(\mathbb {R})$ and $\mathrm {vol} \, \mathbb {G}_{n,k}$ the Riemannian volumes of the orthogonal group and Grassmannian, respectively, and note the relation

(2.1) $$ \begin{align} \mathrm{vol} \,\mathbb{G}_{n,k}=\frac{\mathrm{vol} \operatorname{\mathrm{O}}_n(\mathbb{R})}{\mathrm{vol} \operatorname{\mathrm{O}}_{k}(\mathbb{R})\cdot\mathrm{vol} \operatorname{\mathrm{O}}_{n-k}(\mathbb{R})}. \end{align} $$

Define the constant

(2.2) $$ \begin{align} c_{n,k}=\frac{\mathrm{vol} \operatorname{\mathrm{O}}_{k}(\mathbb{R})\cdot\mathrm{vol} \operatorname{\mathrm{O}}_{n-k}(\mathbb{R})}{\binom nk}. \end{align} $$

Theorem 2. For any $A\in \operatorname {\mathrm {GL}}_n(\mathbb {R})$ , we have

$$ \begin{align*} \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}} \Big[\!\!\sup_{\substack{g\in\mathbb{G}_{n,k}:\\ (UA)_\#g=g}}(\log^+|{\det}\ UA|_g|)\Big]d{\operatorname{\mathrm{O}}_n(\mathbb{R})} \geq c_{n,k}\int_{g\in\mathbb{G}_{n,k}}\log^+|{\det}\ A|_g|\,d\mathbb{G}_{n,k}. \end{align*} $$

If we integrate instead with respect to the Haar measure $dU$ on $\operatorname {\mathrm {O}}_n(\mathbb {R})$ and the invariant probability measure $dg$ on $\mathbb {G}_{n,k}$ , which is just $d\mathbb {G}_{n,k}$ normalized to have volume one, we get

(2.3) $$ \begin{align} \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}} \bigg[\sup_{\substack{g\in\mathbb{G}_{n,k}:\\ (UA)_\#g=g}}(\log^+|{\det}\ UA|_g|)\bigg]dU \geq \frac{1}{\binom nk} \int_{g\in\mathbb{G}_{n,k}}\log^+|{\det}\ A|_g|\,dg. \end{align} $$

This follows immediately from Theorem 2 and equation (2.1). The proof of Theorem 2 is given in §§2 and 5.

Note that Theorem 2 implies a slightly more general result.

Theorem 3. If $\mu $ is an orthogonally invariant probability measure on $\operatorname {\mathrm {GL}}_n(\mathbb {R})$ , then

$$ \begin{align*} \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})} \Big[\sup_{\substack{g\in \mathbb{G}_{n,k}:\\A_{\#}g=g}}\log^{+}|{\det}\ A|_g|\Big]d\mu \geq \frac{1}{\binom{n}{k}}\int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})} \int_{g\in \mathbb{G}_{n,k}} \log^{+}|{\det}\ A|_g|\,dg\,d\mu. \end{align*} $$

Proof. Since $\mu $ is an orthogonally invariant probability measure on $\operatorname {\mathrm {GL}}_n(\mathbb {R})$ , for every integrable function $\eta :\operatorname {\mathrm {GL}}_n(\mathbb {R})\to \mathbb {R}$ , we have

(2.4) $$ \begin{align} \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})} \eta(UA)\,d\mu = \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})} \eta (A)\,d\mu. \end{align} $$

(This is just the change of variable formula of measure theory for the transformation $T_U:\operatorname {\mathrm {GL}}_n(\mathbb {R})\to \operatorname {\mathrm {GL}}_n(\mathbb {R})$ given by $T_U(A)=UA$ . Then, by the ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ -invariance of $\mu $ , we have that the pushforward measure $(T_U)_*\mu $ coincides with $\mu $ .)

For short, let us define $\varphi :\operatorname {\mathrm {GL}}_n(\mathbb {R})\to \mathbb {R}$ by

$$ \begin{align*} \varphi(B)=\sup_{\substack{g\in\mathbb{G}_{n,k}:\\ (B)_\#g=g}}(\log^+|{\det}\ B|_g|). \end{align*} $$

Then integrating over $\operatorname {\mathrm {GL}}_n(\mathbb {R})$ , with respect to $\mu $ , on both sides of the inequality of Theorem 2, we obtain

$$ \begin{align*} &\int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\bigg\{\! \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}} \varphi(UA) d{\operatorname{\mathrm{O}}_n(\mathbb{R})} \bigg\}\,d\mu\\&\quad\geq c_{n,k}\int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\int_{g\in\mathbb{G}_{n,k}}\log^+|{\det}\ A|_g|\,d\mathbb{G}_{n,k}\,d\mu. \end{align*} $$

Applying Fubini on the left-hand side,

$$ \begin{align*} \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\bigg\{\! \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}} \varphi(UA) d{\operatorname{\mathrm{O}}_n(\mathbb{R})} \bigg\}\,d\mu &= \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}}\bigg\{\! \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})} \varphi(UA)d\mu \bigg\}\,d{\operatorname{\mathrm{O}}_n(\mathbb{R})} \\ & =\mathrm{vol} \,{\operatorname{\mathrm{O}}_n(\mathbb{R})} \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\varphi(A) \,d\mu, \end{align*} $$

where the last equality follows from the ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ -invariance of $\mu $ as in equation (2.4). Using the facts that

$$ \begin{align*} \frac{c_{n,k}}{\mathrm{vol} \,{\operatorname{\mathrm{O}}_n(\mathbb{R})}}=\frac{1}{\binom nk\mathrm{vol} \, \mathbb{G}_{n,k}} \quad\mbox{and}\quad d\mathbb{G}_{n,k}=\mathrm{vol} \,\mathbb{G}_{n,k}\cdot dg \end{align*} $$

completes the proof.

Proof of Theorem 1

Pointwise, we have

where the supremum on the right-hand side is defined to be $0$ if the set of $g\in \mathbb {G}_{n,k}$ such that $A_\#g=g$ is empty.

Then, for finishing the proof of Theorem 1, it suffices to identify the right-hand side of the expression in Theorem 3 in terms of $(\sum _{i=1}^kr_i)^+$ . As in the proof of [Reference Dedieu and ShubDS03, Theorem 3],

$$ \begin{align*} \sum_{i=1}^k r_i =\int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\int_{g\in \mathbb{G}_{n,k}}\log |{\det}\ A|_g|\, dg\,d\mu, \end{align*} $$

so

$$ \begin{align*} \bigg(\!\sum_{i=1}^k r_i\bigg)^+ &= \bigg(\!\int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\int_{g\in \mathbb{G}_{n,k}}\log |{\det}\ A|_g|\, dg\,d\mu\bigg)^+\\ & \leq \int_{A\in\operatorname{\mathrm{GL}}_n(\mathbb{R})}\int_{g\in \mathbb{G}_{n,k}}\log^+ |{\det}\ A|_g|\, dg\,d\mu.\\[-3.6pc] \end{align*} $$

We will give the proof of Theorem 2 in §§2 and 5 after some preparations in the next section.

3 Manifold of fixed subspaces

Let $A\in \operatorname {\mathrm {GL}}_n(\mathbb {R})$ , and define the manifold of fixed k-dimensional subspaces

$$ \begin{align*} {\mathbb V}_A:=\{(U,g)\in {\operatorname{\mathrm{O}}_n(\mathbb{R})}\times \mathbb{G}_{n,k}: (UA)_{\#}g=g \}. \end{align*} $$

Let $\Pi _1:{\mathbb V}_A\to {\operatorname {\mathrm {O}}_n(\mathbb {R})}$ and $\Pi _2:{\mathbb V}_A\to \mathbb {G}_{n,k}$ be the associated projections.

Given $g\in \mathbb {G}_{n,k}$ , one has

$$ \begin{align*} \Pi_2^{-1}(g)=\{(U,g): U\in {\operatorname{\mathrm{O}}_n(\mathbb{R})},\, A_{\#}g=(U^{-1})_{\#}g\}. \end{align*} $$

By abusing notation, we identify $\Pi _2^{-1}(g)$ with $\Pi _1\Pi _2^{-1}(g)$ , which we in turn identify with $\operatorname {\mathrm {O}}(k)\times \operatorname {\mathrm {O}}(n-k)$ . Similarly, given $U\in {\operatorname {\mathrm {O}}_n(\mathbb {R})}$ , we identify $ \Pi _1^{-1}(U)$ with

$$ \begin{align*}\{g\in \mathbb{G}_{n,k}:\mbox{fixed by } (UA)_\#\}. \end{align*} $$

The tangent space to the Grassmannian $\mathbb {G}_{n,k}$ at g can be identified in a natural way with the set of linear maps $\mathrm {Hom}(g,g^\perp )$ , that is, any subspace $g'\in \mathbb {G}_{n,k}$ in a neighborhood of g can be represented as the graph of a unique map in $\mathrm {Hom}(g,g^\perp )$ . More precisely, if we denote by $\pi _g$ and $\pi _{g^\perp }$ the orthogonal projections of $\mathbb {R}^n=g\oplus g^\perp $ into g and $g^\perp $ , respectively, then $g'\in \mathbb {G}_{n,k}$ such that $g'\cap g^\perp =\{0\}$ is the graph of the linear map $\pi _{g^\perp }\circ ((\pi _{g})|_{g'})^{-1}$ .

Lemma 3. Let $B\in \operatorname {\mathrm {GL}}_n(\mathbb {R})$ and $g\in \mathbb {G}_{n,k}$ such that $B_\#g=g$ . Then, the induced map $\mathcal {L}_B:\mathrm {Hom}(g,g^\perp )\to \mathrm {Hom}(g,g^\perp )$ , on local charts, is given by

$$ \begin{align*} \mathcal{L}_B(\varphi)= [\pi_{g^\perp}(B|_{g^\perp})]\circ \varphi\circ ([\pi_{g}(B|_{g})]+ [\pi_{g}(B|_{g^\perp})]\circ \varphi) )^{-1}. \end{align*} $$

Furthermore, its derivative at g, represented by $0\in \mathrm {Hom}(g,g^\perp )$ , is given by

$$ \begin{align*} D\mathcal{L}_B(0)\dot\varphi= [\pi_{g^\perp}(B|_{g^\perp})]\circ\dot\varphi\circ [\pi_{g}(B|_{g})]^{-1}. \end{align*} $$

Let us denote by $\mathrm {NJ}_{\Pi _1}$ and $\mathrm {NJ}_{\Pi _2}$ the normal Jacobians of the maps $\Pi _1$ and $\Pi _2$ , respectively, where the normal Jacobian of a surjective linear map $L:V_1\to V_2$ of finite dimensional real vector spaces with inner product is the absolute value of the determinant of the linear map L restricted to the orthogonal complement of the kernel of L in $V_1$ . (See [Reference Dedieu and ShubDS03, §3.1].)

In §5, we will need the normal Jacobian written more explicitly. To this end, choose bases $v_1, \ldots , v_k$ for g and $v_{k+1}, \ldots , v_n$ for its orthogonal complement $g^\perp $ . In terms of the basis $v_1,\ldots , v_n$ of $\mathbb {R}^n$ , a linear map $B:\mathbb {R}^n\to \mathbb {R}^n$ which satisfies $Bg=g$ is represented by a matrix of the form

$$ \begin{align*} \left(\begin{array}{ c | c } B_1 & * \\ \hline 0 & B_2 \end{array}\right)\hspace{-1.5pt}. \end{align*} $$

By Lemma 3, if X is the matrix representing $\dot \varphi $ in this basis, then $D\mathcal {L}_B(0)\dot \varphi $ is represented by the matrix $B_2XB_1^{-1}$ .

Lemma 5. Let $(U,g)\in \mathbb V_A$ and let

$$ \begin{align*} \left(\begin{array}{ c | c } B_1 & * \\ \hline 0 & B_2 \end{array}\right) \end{align*} $$

represent the map $UA$ in the basis $v_1,\ldots , v_n$ defined above. Then,

$$ \begin{align*}\det (\mathrm{Id} -D\mathcal{L}_{UA}(g))= \det(\mathrm{Id}-B_2\otimes \,{\vphantom{B}}^{t}\!{B}_1^{-1}). \end{align*} $$

4 Proof of Theorem 2

Let $\phi :\mathbb {G}_{n,k}\to \mathbb {R}$ be an integrable function, and let $\hat \phi :\mathbb {V}_A\to \mathbb {R}$ be its lift to $\mathbb V_A$ , that is, $\hat \phi $ is given by $\hat \phi :=\phi \circ \Pi _2$ . (Note that given $g\in \mathbb {G}_{n,k}$ , $\hat \phi $ is constant in the fiber $\Pi _2^{-1}(g)$ , and its value coincides with the value of $\phi $ at g.)

For a set of full measure of $U\in {\operatorname {\mathrm {O}}_n(\mathbb {R})}$ (cf. Remark 2), we have

(4.1) $$ \begin{align} \binom{n}{k}\sup_{g\in\Pi_1^{-1}(U)}(\phi(g))\geq \#\Pi_1^{-1}(U)\sup_{g\in\Pi_1^{-1}(U)}(\phi(g))\geq \sum_{g\in\Pi_1^{-1}(U)}\phi(g). \end{align} $$

By the coarea formula applied to $\Pi _1$ , we get

(4.2) $$ \begin{align} \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}}\bigg(\!\sum_{g\in\Pi_1^{-1}(U)}\phi(g)\bigg)\,d{\operatorname{\mathrm{O}}_n(\mathbb{R})}= \int_{{\mathbb V}_A}\hat\phi(U,g)\, \mathrm{NJ}_{\Pi_1}(U,g)\, d{\mathbb V}_A. \end{align} $$

However, applying the coarea formula to the projection $\Pi _2$ ,

(4.3) $$ \begin{align} &\int_{{\mathbb V}_A}\hat\phi(U,g)\, \mathrm{NJ}_{\Pi_1}(U,g)\, d{\mathbb V}_A \\& \quad =\int_{g\in\mathbb{G}_{n,k}}\bigg(\!\int_{U\in \Pi_2^{-1}(g)} \phi(g)\, \mathrm{NJ}_{\Pi_1}(U,g)\, d\Pi_2^{-1}(g) \bigg)\,d\mathbb{G}_{n,k},\nonumber \end{align} $$

where we have used the fact that $\mathrm {NJ}_{\Pi _2}=1$ , and $d\Pi _2^{-1}(g)$ is the volume form on $\Pi _2^{-1}(g)$ induced by the restriction of the Riemannian metric on ${\mathbb V}_A$ to $\Pi _2^{-1}(g)$ .

Then from equations (4.1), (4.2), (4.3), and Lemma 4, we have

(4.4) $$ \begin{align} & \int_{U\in{\operatorname{\mathrm{O}}_n(\mathbb{R})}} \Big(\sup_{g\in\Pi_1^{-1}(U)}(\phi(g))\Big)\,d{\operatorname{\mathrm{O}}_n(\mathbb{R})}\nonumber\\ \nonumber & \quad \geq \binom nk^{-1} \int_{g\in\mathbb{G}_{n,k}}\phi(g)\bigg[\!\int_{U\in \Pi_2^{-1}(g)} {\mathrm{NJ}_{\Pi_1}(U,g)}\, d\Pi_2^{-1}(g) \bigg]\,d\mathbb{G}_{n,k}\\ & \quad =\binom nk^{-1} \int_{g\in\mathbb{G}_{n,k}}\phi(g)\bigg[\!\int_{U\in \Pi_2^{-1}(g)}|{\det}\ \mathrm{Id} -D\mathcal{L}_{UA}(g)| \, d\Pi_2^{-1}(g) \bigg]\,d\mathbb{G}_{n,k}. \end{align} $$

Specialize now to $\phi :\mathbb {G}_{n,k}\to \mathbb {R}$ given by

$$ \begin{align*} \phi(g):=\log^+ |{\det}\ A|_g|,\quad g\in \mathbb{G}_{n,k}. \end{align*} $$

In particular,

$$ \begin{align*}\sup_{g\in{\Pi}_1^{-1}(U)}\phi(g) = \sup_{\substack{g\in{\mathbb{G}}_{n,k}:\\(UA)_\#g=g}}\log^+ |{\det}\ (UA)|_g|. \end{align*} $$

Now, the proof of Theorem 2 follows from Theorem 4 below which is used to bound the bracketed inner integral in equation (4.4); this together with the non-negativity of $\phi $ proves Theorem 2.

Theorem 4. Given $g\in \mathbb {G}_{n,k}$ , one has

$$ \begin{align*} \int_{U\in \Pi_2^{-1}(g)} ( \det \mathrm{Id} -D\mathcal{L}_{UA}(g))\, d\Pi_2^{-1}(g)(U) \geq \mathrm{vol} \operatorname{\mathrm{O}}_k(\mathbb{R})\cdot\mathrm{vol} \operatorname{\mathrm{O}}_{n-k}(\mathbb{R}). \end{align*} $$

The proof is given in the following section.

5 Proof of Theorem 4

For fixed $g\in \mathbb {G}_{n,k}$ , choose $U_0\in {\operatorname {\mathrm {O}}_n(\mathbb {R})}$ such that $U_0Ag=g$ . Then,

$$ \begin{align*}\Pi_2^{-1}(g)=\{VU_0:V\in{\operatorname{\mathrm{O}}_k(\mathbb{R})}\times{\operatorname{\mathrm{O}}_{n-k}(\mathbb{R})} \},\end{align*} $$

where we continue to identify ${\operatorname {\mathrm {O}}_k(\mathbb {R})}\times {\operatorname {\mathrm {O}}_{n-k}(\mathbb {R})}$ with $\operatorname {\mathrm {O}}(g)\times \operatorname {\mathrm {O}}(g^\perp ) $ . We have

$$ \begin{align*} &\int_{U\in \Pi_2^{-1}(g)} \det( \mathrm{Id} -D\mathcal{L}_{UA}(g))\, d\Pi_2^{-1}(g)(U) \\ &\quad= \int_{V\in{\operatorname{\mathrm{O}}_k(\mathbb{R})}\times{\operatorname{\mathrm{O}}_{n-k}(\mathbb{R})}}\det(\mathrm{Id}-D\mathcal{L}_{VU_0A}(g))\, d\Pi_2^{-1}(g)(VU_0)\\ &\quad=\mathrm{vol} \operatorname{\mathrm{O}}_k(\mathbb{R})\cdot\mathrm{vol} \operatorname{\mathrm{O}}_{n-k}(\mathbb{R})\\ & \qquad\times\int_{\psi_1\in{\operatorname{\mathrm{O}}_k(\mathbb{R})}}\int_{\psi_2\in{\operatorname{\mathrm{O}}_{n-k}(\mathbb{R})}} \det(\mathrm{Id}-(\psi_2B_2)\otimes\,{\vphantom{(}}^{t}\!{(}\psi_1B_1)^{-1})\, d\psi_2\,d\psi_1, \end{align*} $$

where $d\psi _1,d\psi _2$ are the Haar measures on ${\operatorname {\mathrm {O}}_k(\mathbb {R})}$ and ${\operatorname {\mathrm {O}}_{n-k}(\mathbb {R})}$ . The last equality follows from Lemma 5, with

$$ \begin{align*} B_1=\pi_g((U_0A)|_g)\quad\mbox{and}\quad B_2=\pi_{g^\perp}((U_0A)|_{g^\perp}). \end{align*} $$

More generally, for $B_1\in \operatorname {\mathrm {GL}}_k(\mathbb {R}), B_2\in \operatorname {\mathrm {GL}}_{n-k}(\mathbb {R})$ , we consider the integral of the characteristic polynomial expressed in the real variable u:

(5.1) $$ \begin{align} {\mathcal {J}}(B_1,B_2;u)=\int_{\psi_1\in {\operatorname{\mathrm{O}}_k(\mathbb{R})}}\int_{\psi_2\in {\operatorname{\mathrm{O}}_{n-k}(\mathbb{R})}} \det(\mathrm{Id}-u(\psi_2B_2)\otimes\,{\vphantom{(\psi_1 B_1)^{-1}}}^{t}\!{(\psi_1 B_1)^{-1}})\,d\psi_2\,d\psi_1. \end{align} $$

Therefore, Theorem 4 is equivalent to

(5.2) $$ \begin{align} {\mathcal {J}}(B_1,B_2;1)\geq 1. \end{align} $$

In fact, we will prove an explicit formula for the integral, expressing the coefficients of the characteristic polynomial ${\mathcal {J}}(B_1,B_2;u)$ as polynomials in the squares of the singular values of $B_1$ and $B_2^{-1}$ with positive integer coefficients.

We complete the proof of Theorem 4 and the inequality in equation (5.2) in several steps. First, we use the representation theory of the general linear group to decompose the double integral into a linear combination of a product of two integrals over ${\operatorname {\mathrm {O}}_n(\mathbb {R})}$ and ${\operatorname {\mathrm {O}}_{n-k}(\mathbb {R})}$ , respectively. Next, each orthogonal group integral is identified with a spherical polynomial. Finally, the theorem follows from an identity between spherical polynomials and Jack polynomials, and a positivity result for the latter due to Knop and Sahi [Reference Knop and SahiKS97]. We first review some notation and terminology from combinatorics and representation theory.

5.1 Preliminaries

Let $\unicode{x3bb} =(\unicode{x3bb} _1, \unicode{x3bb} _2,\ldots , \unicode{x3bb} _k)$ be an integer partition of n with k parts:

(5.3)

Associated with the partition $\unicode{x3bb} $ is a Young diagram which is a left justified arrangement of n boxes into k rows, with $\unicode{x3bb} _i$ boxes in the ith row. For example, for the partition $\unicode{x3bb} =(5,3,1)$ of 9 into three parts, the associated Young diagram is

The conjugate partition to $\unicode{x3bb} $ , denoted $\unicode{x3bb} '$ , is obtained by interchanging the rows and columns of the Young diagram of $\unicode{x3bb} $ . For the partition $\unicode{x3bb} =(5,3,1)$ depicted above, we have $\unicode{x3bb} '=(3,2,2,1,1).$

Partitions $\unicode{x3bb} $ with at most n-parts—or equivalently, Young diagrams with at most n rows—parameterize irreducible polynomial representations of $G=GL_n(\mathbb {R})$ . For example, letting $V_0$ be the standard n-dimensional representation of G, the partition $(r)$ corresponds to $\operatorname {\mathrm {sym}}^r(V_0)$ and $(1,1,\ldots , 1)$ (with r ones) corresponds to $\Lambda ^r(V_0).$ More generally, letting $a_i$ be the number of columns of length i in the Young diagram of $\unicode{x3bb} ,$ the irreducible representation corresponding to $\unicode{x3bb} $ can be identified with a subspace of

(5.4) $$ \begin{align} \operatorname{\mathrm{sym}}^{a_n}\Lambda^n (V_0)\otimes \operatorname{\mathrm{sym}}^{a_{n-1}}\Lambda^{n-1} (V_0)\otimes \cdots \otimes \operatorname{\mathrm{sym}}^{a_1} V_0. \end{align} $$

The precise definition of this irreducible representation is not relevant for our present concerns. However, we note that the representation corresponding to $\unicode{x3bb} $ has a vector fixed by the orthogonal group $\operatorname {\mathrm {O}}_n(\mathbb {R})$ if and only if every part of $\unicode{x3bb} $ is even. (See §5.5.2 for an example.) This observation, presented in Theorem 6 below, and the more explicit positivity statement of Theorem 5 are the key ideas in our proof of Theorem 4.

5.2 Orthogonal group integrals

We begin by expanding the characteristic polynomial in the integrand as a sum of traces:

$$ \begin{align*} \det(\mathrm{Id}-u(\psi_2 B_2)\otimes\,{\vphantom{(\psi_1 B_1)^{-1}}}^{t}\!{(\psi_1 B_1)^{-1}}) =\sum_{j=0}^{k(n-k)} (-u)^j \operatorname{\mathrm{tr}}\bigwedge\nolimits^j(\psi_2 B_2\otimes \,{\vphantom{(}}^{t}\!{(}\psi_1 B_1)^{-1}). \end{align*} $$

Next, decompose the exterior powers of the tensor product as

(5.5) $$ \begin{align} \bigwedge\nolimits^j(\psi_2 B_2\otimes \,{\vphantom{(}}^{t}\!{(}\psi_1 B_1)^{-1}) = \sum_{\unicode{x3bb}:|\unicode{x3bb}|=j} \rho_{\unicode{x3bb}'}(\psi_2 B_2)\otimes \rho_{\unicode{x3bb}}(\,{\vphantom{(}}^{t}\!{(}\psi_1 B_1)^{-1}), \end{align} $$

where:

• • the sum is over all partitions $\unicode{x3bb} $ of j with at most k rows and $n-k$ columns;

• • $\unicode{x3bb} '$ is the partition conjugate to $\unicode{x3bb} $ ; and

• • $\rho _\unicode{x3bb} , \rho _{\unicode{x3bb} '}$ are the irreducible representations of $\operatorname {\mathrm {GL}}_k(\mathbb {R})$ and $\operatorname {\mathrm {GL}}_{n-k}(\mathbb {R})$ associated to the partitions $\unicode{x3bb} ,\unicode{x3bb} ',$ respectively.

See, for example, [Reference Fulton and HarrisFH91, Exercise 6.11]. Since the trace of a tensor product of two matrices is the product of the two traces, we may write

$$ \begin{align*} \det(\mathrm{Id}-u(\psi_2 B_2)\otimes\,{\vphantom{(\psi_1 B_1)^{-1}}}^{t}\!{(\psi_1 B_1)^{-1}}) =\sum_{j=0}^{k(n-k)} (-u)^j\sum_{|\unicode{x3bb}|=j} \operatorname{\mathrm{tr}}\rho_{\unicode{x3bb}'}(\psi_2 B_2)\cdot\operatorname{\mathrm{tr}}\rho_\unicode{x3bb} (\psi_1 B_1). \end{align*} $$

Integrating over $\operatorname {\mathrm {O}}_k(\mathbb {R})\times \operatorname {\mathrm {O}}_{n-k}(\mathbb {R})$ , we find that ${\mathcal {J}}(B_1,B_2;u)$ is equal to

(5.6) $$ \begin{align} &\sum_{j=0}^{k(n-k)}(-u)^j \sum_{|\unicode{x3bb}|=j} \bigg(\!\int_{\psi_2\in \operatorname{\mathrm{O}}_{n-k}(\mathbb{R})}\operatorname{\mathrm{tr}}\rho_{\unicode{x3bb}'}(\psi_2 B_2)\,d\psi_2\bigg)\cdot \bigg(\!\int_{\psi_1\in \operatorname{\mathrm{O}}_{k}(\mathbb{R})} \operatorname{\mathrm{tr}}\rho_{\unicode{x3bb}}(\psi_1 B_1)\,d\psi_1\bigg) \nonumber\\ &\qquad= 1+ \sum_{\substack{1\leq j\leq k(n-k)\\ |\unicode{x3bb}| = j}} (-u)^j F_{\unicode{x3bb}'}(B_2)F_{\unicode{x3bb}}(B_1) , \end{align} $$

where for $M\in \operatorname {\mathrm {GL}}_N(\mathbb {R})$ and $\mu $ a partition of j with at most N parts, we define

(5.7) $$ \begin{align} F_\mu(M)=\int_{\psi\in \operatorname{\mathrm{O}}_N(\mathbb{R})}\operatorname{\mathrm{tr}}\rho_\mu(\psi M)\,d\psi. \end{align} $$

Theorem 4 follows from the following more explicit result.

Theorem 5. Let $M\in \operatorname {\mathrm {GL}}_N(\mathbb {R})$ and $\mu =(\mu _1, \ldots , \mu _r)$ with $\mu _1\geq \mu _2\geq \cdots \geq \mu _r>0$ be a partition of k of at most N parts.

1. (1) If any of the parts $\mu _i$ is odd, then $F_\mu (M)=0.$

2. (2) If all the parts $\mu _i$ are even, then $F_\mu (M)$ is an even polynomial in the singular values of M with positive coefficients.

5.3 Spherical polynomials

The proof of Theorem 5 involves the theory of spherical polynomials for the symmetric space $G/K$ , where $G=\operatorname {\mathrm {GL}}_N(\mathbb {R})$ and $K=\operatorname {\mathrm {O}}_N(\mathbb {R})$ , and Jack polynomials. We recall these briefly.

Let ${\mathcal {P}}_N$ be the set of partitions with at most N parts, thus

$$ \begin{align*} {\mathcal {P}}_N=\{\mu \in \mathbb Z^N\mid \mu_1\ge \mu_2\ge \cdots\ \geq\mu_N\ge 0\}. \end{align*} $$

For $\mu \in {\mathcal {P}}_N$ , let $(\rho _\mu , V_\mu )$ be the corresponding representation of G, and let $(\rho _\mu ^\prime ,V_\mu ^*)$ be the contragredient representation. That is, G acts on the dual vector space $V_\mu ^*$ by

$$ \begin{align*}\langle\rho'(g)u,v\rangle=\langle u, \rho(g)^{-1}v\rangle, \end{align*} $$

where $\langle u,v\rangle $ is the evaluation pairing between $u\in V_{\mu }^*$ and $u\in V_\mu .$ A matrix coefficient of $V_\mu $ is a function on G of the form

$$ \begin{align*} \phi_{u,v}(M) =\langle u, \rho_\mu(M)v \rangle, \end{align*} $$

where $u\in V_\mu ^*$ and $v\in V_\mu $ . We write ${\mathcal {F}}_\mu $ for the span of matrix coefficients of $V_\mu $ . Then ${\mathcal {F}}_\mu $ is stable under left and right multiplication by G, and one has a $G\times G$ -module isomorphism

$$ \begin{align*} V_\mu^*\otimes V_\mu \approx {\mathcal {F}}_\mu,\quad u\otimes v\mapsto \phi_{u,v}.\end{align*} $$

Theorem 6. Let $\mu $ be a partition in ${\mathcal {P}}_N$ . Then the following are equivalent:

1. (1) $\mu $ is even, that is, $\mu _i\in 2\mathbb Z$ for all i;

2. (2) $V_\mu $ has a spherical vector, that is, a vector fixed by K;

3. (3) $V_\mu ^*$ has a spherical vector;

4. (4) ${\mathcal {F}}_\mu $ contains a spherical polynomial $\phi _\mu $ , that is, a function satisfying

   $$ \begin{align*} \phi_\mu(kgk')=\phi_\mu(g), \; g\in G,\; k,k'\in K.\end{align*} $$

The spherical vector $v_\mu $ and spherical polynomial $\phi _\mu $ are unique up to scalar multiple, and the latter is usually normalized by the requirement $\phi _\mu (e)=1$ , which fixes it uniquely.

Proof. This follows from the Cartan–Helgason theory of spherical representations [Reference HelgasonHel84, Theorem V.4.1].

We now connect the polynomial $F_\mu $ to $\phi _\mu $ .

Theorem 7. Let $F_\mu (M)$ be as in equation (5.7). If $\mu $ is even, then $F_\mu =\phi _\mu $ , otherwise $F_\mu =0$ .

Proof. If $\{v_i\},\{u_i\}$ are dual bases for $V_\unicode{x3bb} ,V_\unicode{x3bb} ^*$ , then $\operatorname {\mathrm {tr}} \rho _\mu (M)=\sum _i\phi _{u_i,v_i}(M)$ , thus the character $\chi _\mu (M) = \operatorname {\mathrm {tr}} \rho _\mu (M)$ is an element of ${\mathcal {F}}_\mu $ . Since ${\mathcal {F}}_\mu $ is stable under the left action of K, it follows that $F_\mu (M)=\int _K\chi _\mu (k M)\,dk$ is in ${\mathcal {F}}_\mu $ as well.

We next argue that $F_\mu $ is $K\times K$ invariant. For this, we compute as follows:

$$ \begin{align*} F_\mu(k_1Mk_2)=\int_K\chi_\mu(kk_1Mk_2)\,dk = \int_K\chi_\mu(k_2kk_1M)\,dk = F_\mu(M).\end{align*} $$

Here, the first equality holds by definition, the second is a consequence the invariance of the trace character, $\chi _\mu (AB)=\chi _\mu (BA)$ , and the final equality follows from the $K\times K$ invariance of the Haar measure $dk$ .

By Theorem 6, this proves that $F_\mu $ is a multiple of $\phi _\mu $ if $\mu $ is even, and $F_\mu =0$ otherwise. To determine the precise multiple, we need to compute the following integral for even $\mu $ :

$$ \begin{align*} F_\mu(e)=\int_K\chi_\mu(k)\,dk.\end{align*} $$

By Schur orthogonality, this integral is the multiplicity of the trivial representation in the restriction of $V_\mu $ to K, which is $1$ if $\mu $ is even. Thus, we get $F_\mu =\phi _\mu $ , as desired.

5.4 Jack polynomials

Jack polynomials $J_\unicode{x3bb} ^{(\alpha )}(x_1,\ldots ,x_N)$ are a family of symmetric polynomials in N variables whose coefficients depend on a parameter $\alpha $ . The main result of [Reference Knop and SahiKS97] is that these coefficients are themselves positive integral polynomials in the parameter $\alpha $ .

Spherical functions correspond to Jack polynomials with $\alpha =2$ . More precisely, we have

(5.8) $$ \begin{align} \phi_\mu(g)= \frac{J_\unicode{x3bb}^{(2)}(a_1,\ldots, a_N)}{J_\unicode{x3bb}^{(2)}(1,\ldots, 1)}, \quad \mu=2\unicode{x3bb}, \end{align} $$

where $a_1,\ldots ,a_N$ are the eigenvalues of the symmetric matrix $\,{\vphantom {g}}^{t}\!{g}g$ ; in other words, the $a_i$ are the squares of the singular values of g.

We can now finish the proof of Theorem 5.

Proof of Theorem 5

Part (1) follows from Theorem 7. Part (2) follows from equation (5.8) and the positivity of Jack polynomials as proved in [Reference Knop and SahiKS97].

5.5 Examples

We conclude this section with two low rank examples of the characteristic polynomials ${\mathcal {J}}(A,B;u)$ for $A\in \operatorname {\mathrm {GL}}_k(\mathbb {R}), B\in \operatorname {\mathrm {GL}}_{n-k}(\mathbb {R}).$ As we may assume A and B are diagonal, let us write

$$ \begin{align*}A=\operatorname{\mathrm{diag}}(a_1,\ldots, a_k)\quad\mbox{and}\quad B=\operatorname{\mathrm{diag}}(b_1, \ldots, b_{n-k}). \end{align*} $$

5.5.1 The case $n=4,k=2$

Here we consider the integral

(5.9) $$ \begin{align} {\mathcal {J}}(A,B;u)=\int_{\psi_1\in \operatorname{\mathrm{O}}_2(\mathbb{R})}\int_{\psi_2\in \operatorname{\mathrm{O}}_2(\mathbb{R})} \det(\mathrm{Id}-u(\psi_2B)\otimes\,{\vphantom{(\psi_1 A)^{-1}}}^{t}\!{(\psi_1 A)^{-1}})\,d\psi_2\,d\psi_1. \end{align} $$

As we are essentially integrating over the circle, it is easy to compute this directly and see that

(5.10) $$ \begin{align} {\mathcal {J}}(A,B;u)=1+\frac{\det(B)^2}{\det(A)^2}u^4=1+\frac{b_1^2b_2^2}{a_1^2a_2^2}u^4. \end{align} $$

5.5.2 The case $n=6,k=2$

In this case, we use equation (5.6) to compute

$$\begin{align*}{\mathcal {J}}(A,B;u)=\int_{\psi_1\in \operatorname{\mathrm{O}}_2(\mathbb{R})}\int_{\psi_2\in \operatorname{\mathrm{O}}_4(\mathbb{R})} \det(\mathrm{Id}-u(\psi_2B)\otimes\,{\vphantom{(\psi_1 A)^{-1}}}^{t}\!{(\psi_1 A)^{-1}})\,d\psi_2\,d\psi_1 \end{align*}$$

for $A\in \operatorname {\mathrm {GL}}_2(\mathbb {R}),B\in \operatorname {\mathrm {GL}}_4(\mathbb {R}).$ Write

$$ \begin{align*}{\mathcal {J}}(A,B;u)=1+c_2u^2+c_4u^4+c_6u^6+c_8u^8. \end{align*} $$

By part (1) of Theorem 5, we immediately see that $c_2=c_6=0$ because there are no partitions $\unicode{x3bb} $ of $2$ or $6$ for which both $\unicode{x3bb} $ and its conjugate $\unicode{x3bb} '$ have only even parts. The only even partition of $k=8$ with at most two parts and with even conjugate is $\unicode{x3bb} =(4,4).$ For V, the standard two-dimensional representation of $\operatorname {\mathrm {GL}}_2(\mathbb {R})$ , we have that $\rho _\unicode{x3bb} (V)=\operatorname {\mathrm {sym}}^4(\Lambda ^2 V)$ is the fourth power of the determinant representation. Hence,

$$ \begin{align*}F_\unicode{x3bb}(A^{-1})=\det A^{-4}.\end{align*} $$

Similarly for W, the standard four-dimensional representation of $\operatorname {\mathrm {GL}}_4(\mathbb {R})$ , the conjugate $\unicode{x3bb} '=(2,2,2,2)$ and $\rho _{\unicode{x3bb} '}(W)=\operatorname {\mathrm {sym}}^2(\Lambda ^4(W))$ is the square of the determinant. Hence, $F_{\unicode{x3bb} '}(B)=\det B^{2}$ and

$$ \begin{align*}c_8=\frac{\det(B)^2}{\det(A)^4}.\end{align*} $$

The only even partition of $k=4$ with even conjugate is $\unicode{x3bb} =\unicode{x3bb} '=(2,2).$ In this case, $\rho _\unicode{x3bb} (V)$ is the square of the determinant representation. The dimension 20 representation $\rho _\unicode{x3bb} (W)$ is a quotient of $\operatorname {\mathrm {sym}}^2(\Lambda ^2(W))$ with a unique $\operatorname {\mathrm {O}}_4(\mathbb {R})$ -fixed vector, namely, the image of

$$ \begin{align*}v=(e_1\wedge e_2)^2+(e_1\wedge e_3)^2+(e_1\wedge e_4)^2+(e_2\wedge e_3)^2+(e_2\wedge e_4)^2+(e_3\wedge e_4)^2. \end{align*} $$

It is readily seen that the trace $\rho _\unicode{x3bb} (B)$ restricted to the span of v is

$$ \begin{align*}\sum_{1\leq i< j\leq 4} b_i^2b_j^2. \end{align*} $$

Then, including the normalizing factor of $1/J^{(2)}_{(1,1)}(1,1,1,1)=1/6$ , we conclude that

$$ \begin{align*}c_4=\frac{1/6 (b_1^2b_2^2+b_1^2b_3^2+b_1^2b_4^2+b_2^2b_3^2+b_2^2b_4^2+b_3^2b_4^2)} {a_1^2a_2^2}.\end{align*} $$