Hostname: page-component-cb9f654ff-mwwwr Total loading time: 0 Render date: 2025-08-29T13:16:41.628Z Has data issue: false hasContentIssue false

Newell–Littlewood numbers III: Eigencones and GIT-semigroups

Published online by Cambridge University Press:  27 August 2025

Shiliang Gao
Affiliation:
Department of Mathematics, Cornell University, Ithaca, NY 14853, USA sg2573@cornell.edu
Gidon Orelowitz
Affiliation:
Department of Mathematics, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA gidono2@illinois.edu
Nicolas Ressayre
Affiliation:
Université Claude Bernard Lyon 1, ICJ UMR5208, CNRS, Ecole Centrale de Lyon, INSA Lyon, Université Jean Monnet, Villeurbanne 69622, France ressayre@math.univ-lyon1.fr
Alexander Yong
Affiliation:
Department of Mathematics, University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA ayong@illinois.edu
Rights & Permissions [Opens in a new window]

Abstract

The Newell–Littlewood (NL) numbers are tensor product multiplicities of Weyl modules for the classical groups in the stable range. Littlewood–Richardson (LR) coefficients form a special case. Klyachko connected eigenvalues of sums of Hermitian matrices to the saturated LR-cone and established defining linear inequalities. We prove analogues for the saturated NL-cone: a description by Extended Horn inequalities (as conjectured in part II of this series), where, using a result of King’s, this description is controlled by the saturated LR-cone and thereby recursive, just like the Horn inequalities; a minimal list of defining linear inequalities; an eigenvalue interpretation; and a factorization of Newell–Littlewood numbers, on the boundary.

Information

Type
Research Article
Creative Commons
Creative Common License - CCCreative Common License - BY
This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
Copyright
© The Author(s), 2025

1. Introduction

Fix $n\in {\mathbb {N}}:=\{1,2,3,\ldots \}$ . This is the third installment in a series [Reference Gao, Orelowitz and YongGOY20a, Reference Gao, Orelowitz and YongGOY20b] about the Newell–Littlewood (NL) numbers [Reference NewellNew51, Reference LittlewoodLit58]

(1) \begin{align} N_{\lambda ,\mu ,\nu }=\sum _{\alpha ,\beta ,\gamma } c_{\alpha ,\beta }^{\lambda } c_{\beta ,\gamma }^{\mu }c_{\gamma ,\alpha }^{\nu }; \end{align}

the indices are partitions in ${{\operatorname {Par}}}_n=\{(\lambda _1, \lambda _2, \ldots , \lambda _n)\in {{\mathbb {Z}}}_{\geqslant 0}^n: \lambda _1\geqslant \lambda _2\geqslant \cdots \geqslant \lambda _n\}$ . In (1), $c^{\lambda }_{\alpha ,\beta }$ is the Littlewood–Richardson (LR) coefficient. The Littlewood–Richardson coefficients are themselves Newell–Littlewood numbers: if $|\nu |=|\lambda |+|\mu |$ then $N_{\lambda ,\mu ,\nu }=c_{\lambda \mu }^\nu$ . The goal of this series is to establish analogues of results known for Littlewood–Richardson coefficients. This paper proves Newell–Littlewood generalizations of breakthrough results of Klyachko [Reference KlyachkoKly98].

The paper [Reference Gao, Orelowitz and YongGOY20a] investigated

\begin{equation*} {\operatorname {NL-semigroup}}(n)=\{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3\,:\, N_{\lambda ,\mu ,\nu }\gt 0\}. \end{equation*}

Indeed, an $\operatorname {NL-semigroup}$ is a finitely generated semigroup [Reference Gao, Orelowitz and YongGOY20a, § 5.2]. A good approximation of it is the saturated semigroup:

\begin{equation*} {\operatorname {NL-sat}}(n)=\{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n^{\mathbb {Q}})^3\,:\, \exists t\gt 0 \; N_{t\lambda ,t\mu ,t\nu }\neq 0\}, \end{equation*}

where ${\operatorname {Par}}_n^{\mathbb {Q}}=\{(\lambda _1,\ldots ,\lambda _n)\in {\mathbb {Q}}^n\,:\,\lambda _1\geqslant \cdots \geqslant \lambda _n\geqslant 0\}$ . Our main results give descriptions of ${\operatorname {NL-sat}}(n)$ , including with a minimal list of defining linear inequalities.

Fix $m\in {\mathbb {N}}$ and consider the symplectic Lie algebra ${\mathfrak {sp}}(2m,{\mathbb {C}})$ . The irreducible ${\mathfrak {sp}}(2m,{\mathbb {C}})$ -representations $V(\lambda )$ are parametrized by their highest weight $\lambda \in {\operatorname {Par}}_m$ (see § 3.1 for details). The tensor product multiplicities ${\operatorname {mult}}^m_{\lambda ,\mu ,\nu }$ are defined by

\begin{equation*} V(\lambda )\otimes V(\mu )=\sum _{\nu \in {\operatorname {Par}}_m} V(\nu )^{\oplus {\operatorname {mult}}^m_{\lambda ,\mu ,\nu }}. \end{equation*}

Since ${\mathfrak {sp}}(2m,{\mathbb {C}})$ -representations are self-dual, ${\operatorname {mult}}^m_{\lambda ,\mu ,\nu }$ is symmetric in its inputs. The supports of these multiplicities (and more generally when ${\mathfrak {sp}}(2m,{\mathbb {C}})$ is replaced by any semisimple Lie algebra) are of significant interest (see, for example, the survey [Reference KumarKum15] and the references therein). Consider the finitely generated semigroup

\begin{equation*} {{\mathfrak {sp}}\operatorname {-semigroup}}(m)=\{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_m)^3\,:\, {\operatorname {mult}}^m_{\lambda ,\mu ,\nu }\gt 0\}, \end{equation*}

and the cone generated by it,

\begin{equation*} {{\mathfrak {sp}}\operatorname {-sat}}(m)=\{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_m^{\mathbb {Q}})^3\,:\, \exists t\gt 0 \; {\operatorname {mult}}^m_{t\lambda ,t\mu ,t\nu }\gt 0\}. \end{equation*}

For $m\geqslant n$ , by postpending $0$ s, ${\operatorname {Par}}_n$ embeds into ${\operatorname {Par}}_m$ . Newell–Littlewood numbers are tensor product multiplicities for ${\mathfrak {sp}}(2m,{\mathbb {C}})$ in the stable range [Reference Koike and TeradaKT87, Corollary 2.5.3]:

(2) \begin{align} \forall (\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3\quad \mbox {if $m\geqslant 2n$ then } {\operatorname {mult}}^m_{\lambda ,\mu ,\nu }=N_{\lambda ,\mu ,\nu }. \end{align}

Now, (2) immediately implies

(3) \begin{align} {\operatorname {NL-sat}}(n)={{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3 \quad\text{for any}\, m\geqslant 2n. \end{align}

Our first result says the relationship of $\operatorname {NL-sat}$ to ${\mathfrak {sp}}\operatorname {-sat}$ is independent of the stable range.

Theorem 1.1. For any $m\geqslant n\geqslant 1$ ,

\begin{equation*} {\operatorname {NL-sat}}(n)={{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3. \end{equation*}

Theorem 1.1 has a number of consequences. Define

\begin{equation*} {\operatorname {LR-sat}}(n)=\{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n^{\mathbb {Q}})^3\,:\, \exists t\gt 0 \; c_{t\lambda ,t\mu }^{t\nu }\gt 0\}. \end{equation*}

Klyachko [Reference KlyachkoKly98] showed that ${\operatorname {LR-sat}}(n)$ describes the possible eigenvalues $\lambda ,\mu ,\nu$ of two $n\times n$ Hermitian matrices $A,B,C$ (respectively) such that $A+B=C$ . Similarly, Theorem 1.1 shows that ${\operatorname {NL-sat}}(n)$ describes solutions to a more general eigenvalue problem; see § 2.6 and Proposition 3.1.Footnote 1

Another major accomplishment of [Reference KlyachkoKly98] was the first proved description of ${\operatorname {LR-sat}}(n)$ via linear inequalities. We have three such descriptions of ${\operatorname {NL-sat}}(n)$ . We now state the first of these. Set $[n]=\{1,\dots ,n\}$ and $[a,b]= \{a,a+1,\dots ,b\}$ for $a\leqslant b$ . For $A\subset [n]$ and $\lambda \in {\operatorname {Par}}_n$ , let $\lambda _A$ be the partition using the only parts indexed by $A$ ; namely, if $A=\{i_1\lt \cdots \lt i_r\}$ then $\lambda _A=(\lambda _{i_1},\dots ,\lambda _{i_r})$ . In particular, $|\lambda _A|=\sum _{i\in A}\lambda _i$ . Using the known descriptions of ${{\mathfrak {sp}}\operatorname {-sat}}(n)$ [Reference Belkale and KumarBK06, Reference RessayreRes10, Reference RessayreRes12] we deduce from Theorem 1.1 a minimal list of inequalities defining ${\operatorname {NL-sat}}(n)$ .

Theorem 1.2. Let $(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3$ . Then $(\lambda ,\mu ,\nu )\in {\operatorname {NL-sat}}(n)$ if and only if

(4) \begin{align} 0\leqslant |\lambda _A|-|\lambda _{A^{\prime}}|+|\mu _B|-|\mu _{B^{\prime}}|+|\nu _C|-|\nu _{C^{\prime}}| \end{align}

for any subsets $A,A^{\prime},B,B^{\prime},C,C^{\prime}\subset [n]$ such that:

  1. (1) $A \cap A^{\prime}= B \cap B^{\prime} = C \cap C^{\prime} = \emptyset$ ;

  2. (2) $|A|+|A^{\prime}|=|B|+|B^{\prime}|=|C|+|C^{\prime}|=|A^{\prime}|+|B^{\prime}|+|C^{\prime}|=:r$ ;

  3. (3) $c_{\tau ^0(A,A^{\prime})^{\vee [(2n-2r)^r]}\; \tau ^0(B,B^{\prime})^{\vee [(2n-2r)^r]}}^{\tau ^0(C,C^{\prime})}= c_{\tau ^2(A,A^{\prime})^{\vee [r^r]}\; \tau ^2(B,B^{\prime})^{\vee [r^r]}}^{\tau ^2(C,C^{\prime})}=1$ .

Moreover, this list of inequalities is irredundant.

The definition of the partitions occurring in condition (3) is in § 3.2.

The proofs of Theorems 1.1 and 1.2 use ideas of P. Belkale and S. Kumar [Reference Belkale and KumarBK06] on their deformation of the cup product on flag manifolds, as well as the third author’s work on GIT-semigroups/cones [Reference RessayreRes10, Reference RessayreRes12]. We interpret ${\operatorname {NL-sat}}(n)$ from the latter perspective in § 5 (see Proposition 5.2) by study of the truncated tensor cone. Our argument requires us to generalize [Reference Belkale and KumarBK06, Theorem 28] and [Reference RessayreRes10, Theorem B] (recapitulated here as Theorem 2.3); see Theorem 5.1. As an application, we obtain Theorem 1.3 below, which is a factorization of the NL-coefficients on the boundary of ${\operatorname {NL-sat}}(n)$ . Let $\lambda \in {\operatorname {Par}}_n$ and $A,A^{\prime}\subset [n]$ . Write $A^{\prime}=\{i^{\prime}_1\lt \cdots \lt i^{\prime}_s\}$ and $A=\{i_1\lt \cdots \lt i_t\}$ and set

\begin{equation*} \lambda _{A,A^{\prime}}=(\lambda _{i^{\prime}_1},\dots ,\lambda _{i^{\prime}_s},-\lambda _{i_t},\dots ,-\lambda _{i_1}) \quad \text{and}\quad \lambda ^{A,A^{\prime}}=\lambda _{[n]-(A\cup A^{\prime})}, \text{etc}.\end{equation*}

Theorem 1.3. Let $A,A^{\prime},B,B^{\prime},C,C^{\prime}\subset [n]$ such that:

  1. (1) $A \cap A^{\prime}= B \cap B^{\prime} = C \cap C^{\prime} = \emptyset$ ;

  2. (2) $|A|+|A^{\prime}|=|B|+|B^{\prime}|=|C|+|C^{\prime}|=|A^{\prime}|+|B^{\prime}|+|C^{\prime}|=:r$ ;

  3. (3) $c_{\tau ^0(A,A^{\prime})^{\vee [(2n-2r)^r]}\; \tau ^0(B,B^{\prime})^{\vee [(2n-2r)^r]}}^{\tau ^0(C,C^{\prime})}= c_{\tau ^2(A,A^{\prime})^{\vee [r^r]}\; \tau ^2(B,B^{\prime})^{\vee [r^r]}}^{\tau ^2(C,C^{\prime})}=1$ ,

as in Theorem 1.2 . For $(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3$ such that

(5) \begin{align} 0=|\lambda _A|-|\lambda _{A^{\prime}}|+|\mu _B|-|\mu _{B^{\prime}}|+|\nu _C|-|\nu _{C^{\prime}}|, \\[-30pt] \nonumber \end{align}
(6) \begin{align} N_{\lambda ,\mu ,\nu }=c_{\lambda _{A,A^{\prime}},\mu _{B,B^{\prime}}}^{\nu _{C,C^{\prime}}^*} N_{\lambda ^{A,A^{\prime}},\mu ^{B,B^{\prime}},\nu ^{C,C^{\prime}}}. \\[-8pt] \nonumber \end{align}

Theorem 1.3 is analogous to [Reference Derksen and WeymanDW11, Theorem 7.4] and [Reference King, Tollu and ToumazetKTT09, Theorem 1.4] for $c_{\lambda ,\mu }^{\nu }$ .

Knutson and Tao’s celebrated saturation theorem [Reference Knutson and TaoKT99] proves, inter alia, that ${\operatorname {LR-semigroup}}(n)$ is described by Horn’s inequalities (see, for example, Fulton’s survey [Reference FultonFul00]). This posits a generalization.

Conjecture 1.4 (NL-saturation [Reference Gao, Orelowitz and YongGOY20a, Conjecture 5.5]). Let $(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3$ . Then $N_{\lambda ,\mu ,\nu }\neq 0$ if and only if $|\lambda |+|\mu |+|\nu |$ is even and there exists $t\gt 0$ such that $N_{t\lambda ,t\mu ,t\nu }\neq 0$ .

Theorem 1.2 permits us to prove Conjecture 1.4 for $n\leqslant 5$ , by computer-aided calculation of Hilbert bases; see § 6. This is the strongest evidence of the conjecture to date; previously, [Reference Gao, Orelowitz and YongGOY20a, Corollary 5.16] proved the $n=2$ case by combinatorial reasoning.

Let $\lambda _1,\dots ,\lambda _s\in {\operatorname {Par}}_n$ for $s\geqslant 3$ . Treat the indices $1,\dots ,s$ as elements of ${{\mathbb {Z}}}/s{{\mathbb {Z}}}$ . We introduce the multiple Newell–Littlewood number as

(7) \begin{align} N_{\lambda _1,\dots ,\lambda _s}=\sum _{(\alpha _1,\dots ,\alpha _s)\in ({\operatorname {Par}}_n)^{s}}\prod _{i\in {{\mathbb {Z}}}/s{{\mathbb {Z}}}} c_{\alpha _i\,\alpha _{i+1}}^{\lambda _i}. \end{align}

When $s=3$ , we recover the Newell–Littlewood numbers.Footnote 2 Consider the associated semigroup and cone:

\begin{equation*} \operatorname {NL}^s\!\operatorname {-semigroup} (n)=\{(\lambda _1,\dots ,\lambda _s)\in ({\operatorname {Par}}_n)^s\,:\, N_{\lambda _1,\dots ,\lambda _s}\gt 0\} \end{equation*}

and

\begin{equation*} \operatorname {NL}^s\!\operatorname {-sat}(n)=\{(\lambda _1,\dots ,\lambda _s)\in ({\operatorname {Par}}_n^{\mathbb {Q}})^s\,:\, \exists t\gt 0, \ N_{t\lambda _1,\dots ,t\lambda _s}\neq 0\}. \end{equation*}

For any totally ordered set $T = \{t_1\lt \cdots \lt t_m\}$ and $R = \{t_{i_1}\lt \cdots \lt t_{i_r}\}\subseteq T$ , define

(8) \begin{align} \tau (R,T) = (i_r-r\geqslant \cdots \geqslant i_1-1). \end{align}

In most cases, we will be considering some finite $A\subseteq \mathbb {Z}_{\gt 0}$ ; for simplicity, we denote

\begin{equation*} \tau (A):=\tau (A,[n]) \end{equation*}

for sufficiently large $n$ .

The Horn inequalities for ${\operatorname {LR-sat}}(n)$ are recursive, as they depend on ${\operatorname {LR-sat}}(n^{\prime})$ for $n^{\prime}\lt n$ (see [Reference FultonFul00]). Theorem 1.2 is not recursive. However, our next result describes the cone ${\operatorname {NL-sat}}(n)$ by inequalities depending on $\operatorname {NL}^6\!\operatorname {-sat}(n^{\prime})$ for $n^{\prime}\leqslant n$ .

Theorem 1.5. Let $(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n^{{\mathbb {Q}}})^3$ . Then $(\lambda ,\mu ,\nu )\in {\operatorname {NL-sat}}(n)$ if and only if

(9) \begin{align} 0\leqslant |\lambda _A|-|\lambda _{A^{\prime}}|+|\mu _B|-|\mu _{B^{\prime}}|+|\nu _C|-|\nu _{C^{\prime}}| \end{align}

for any subsets $A,A^{\prime},B,B^{\prime},C,C^{\prime}\subset [n]$ such that:

  1. (1) $A \cap A^{\prime}= B \cap B^{\prime} = C \cap C^{\prime} = \emptyset$ ;

  2. (2) $|A|+|A^{\prime}|=|B|+|B^{\prime}|=|C|+|C^{\prime}|=|A^{\prime}|+|B^{\prime}|+|C^{\prime}|=:r$ ;

  3. (3) $(\tau (A), \tau (C^{\prime}), \tau (B), \tau (A^{\prime}), \tau (C), \tau (B^{\prime}))\in \operatorname {NL}^6\!\operatorname {-sat}(r)$ .

This result is proved in § 8.1. By a result of King’s [Reference KingKin71] (see also [Reference Howe, Tan and WillenbringHT05]), each six-fold Newell–Littlewood coefficient is a particular Littlewood–Richardson coefficient (see § 8.1 for details). Consequently, condition (3) is equivalent to checking if some explicitly determined triple of partitions is in ${\operatorname {LR-sat}}(2r)$ . Since ${\operatorname {LR-sat}}(2r)$ is described by the Horn inequalities, we thereby obtain a description of ${\operatorname {NL-sat}}(n)$ only involving inequalities and no tensor product multiplicities. It is in this sense that Theorem 1.5 is of the same spirit as Horn’s original inequalities.

Just as the proof of Horn’s conjecture depends on Knutson and Tao’s saturation theorem, our proof of Theorem 1.5 uses this consequence of King’s result (see § 7.2).

Proposition 1.6 (Six-fold NL-saturation). Let $\lambda _1,\dots ,\lambda _6\in {\operatorname {Par}}_n$ . If there exists a positive integer $t$ such that $N_{t\lambda _1,\dots ,t\lambda _6}\neq 0$ then $N_{\lambda _1,\dots ,\lambda _6}\neq 0$ .

In [Reference Gao, Orelowitz and YongGOY20b, Conjecture 1.4], a conjectural description of ${\operatorname {NL-sat}}(n)$ was given. That conjecture subsumes both Conjecture 1.4 and a description of $\operatorname {NL-sat}$ using extended Horn inequalities [Reference Gao, Orelowitz and YongGOY20b, Definition 1.2]. Theorem 1.5 proves the latter part of the conjecture.

2. Generalities on the tensor cones

2.1 Finitely generated semigroups

A subset $\Gamma \subseteq {{\mathbb {Z}}}^n$ is a semigroup if $\vec 0\in \Gamma$ and $\Gamma$ is closed under addition. A finitely generated semigroup $\Gamma$ generates a closed convex polyhedral cone $\Gamma _{\mathbb {Q}}\subseteq {\mathbb {Q}}^n$ :

\begin{equation*} \Gamma _{\mathbb {Q}}=\{x\in {\mathbb {Q}}^n:\exists t\in {{\mathbb {Z}}}_{\gt 0}\ tx\in \Gamma \}. \end{equation*}

The subgroup of ${{\mathbb {Z}}}^n$ generated by $\Gamma$ is

\begin{equation*} \Gamma _{{\mathbb {Z}}}=\{x-y\,:\,x,y\in \Gamma \}. \end{equation*}

The semigroup $\Gamma$ is saturated if $\Gamma =\Gamma _{{\mathbb {Z}}}\cap \Gamma _{\mathbb {Q}}$ .

2.2 GIT-semigroups

We recall the GIT-perspective of [Reference RessayreRes10]. Let $G$ be a complex reductive group acting on an irreducible projective variety $X$ . Let ${\operatorname {Pic}}^G(X)$ be the group of $G$ -linearized line bundles. Given ${\mathcal L}\in {\operatorname {Pic}}^G(X)$ , let ${\operatorname {H}}^0(X,{\mathcal L})$ be the space of sections of $\mathcal L$ ; it is a $G$ -module. Let ${\operatorname {H}}^0(X,{\mathcal L})^G$ be the subspace of invariant sections. Define

\begin{equation*} {\operatorname {GIT-semigroup}}(G,X)=\{{\mathcal L}\in {\operatorname {Pic}}^G(X)\,:\, {\operatorname {H}}^0(X,{\mathcal L})^G\neq \{0\}\}. \end{equation*}

This is a semigroup since $X$ being irreducible says the product of two nonzero $G$ -invariant sections is a nonzero $G$ -invariant section. The saturated version of it is

\begin{equation*} {\operatorname {GIT-sat}}(G,X)=\{{\mathcal L}\in {\operatorname {Pic}}^G(X)\otimes {\mathbb {Q}}: \exists t\gt 0, \ {\operatorname {H}}^0(X,{\mathcal L}^{\otimes t})^G\neq \{0\}\}. \end{equation*}

2.3 The tensor semigroup

Let $\mathfrak {g}$ be a semisimple complex Lie algebra, with fixed Borel subalgebra $\mathfrak b$ and Cartan subalgebra ${\mathfrak t}\subset {\mathfrak b}$ . Denote by $\Lambda ^+({\mathfrak {g}})\subset {\mathfrak t}^*$ the semigroup of the dominant weights. It is contained in the weight lattice $\Lambda ({\mathfrak {g}})\simeq {{\mathbb {Z}}}^r$ , where $r$ is the rank of $\mathfrak {g}$ . Given $\lambda \in \Lambda ^+({\mathfrak {g}})$ , denote by $V_{\mathfrak {g}}(\lambda )$ (or simply $V(\lambda )$ ) the irreducible representation of $\mathfrak {g}$ with highest weight $\lambda$ . Let $V(\lambda )^*$ be the dual representation. Consider the semigroup

\begin{equation*} {{\mathfrak {g}}\operatorname {-semigroup}}=\{(\lambda ,\mu ,\nu )\in (\Lambda ^+({\mathfrak {g}}))^3\;:\;V(\nu )^*\subset V(\lambda )\otimes V(\mu )\}, \end{equation*}

and the generated cone ${\mathfrak {g}}\operatorname {-sat}$ in $(\Lambda ({\mathfrak {g}})\otimes {\mathbb {Q}})^3$ . When ${\mathfrak {g}}={\mathfrak {sp}}(2m,{\mathbb {C}})$ we have $V(\nu )^*\simeq V(\nu )$ and ${\mathfrak {g}}\operatorname {-semigroup}$ is what we denoted by $\mathfrak {sp}$ -semigroup $(m)$ in the introduction. The set ${\mathfrak {g}}\operatorname {-semigroup}$ spans the rational vector space $(\Lambda ({\mathfrak {g}})\otimes {\mathbb {Q}})^3$ , or equivalently, the cone ${\mathfrak {g}}\operatorname {-sat}$ has nonempty interior. The group $({{\mathfrak {g}}\operatorname {-semigroup}})_{{\mathbb {Z}}}$ is well known (see, for example, [Reference Pasquier and RessayrePR13, Theorem 1.4]):

\begin{equation*} ({{\mathfrak {g}}\operatorname {-semigroup}})_{{\mathbb {Z}}}=\{(\lambda ,\mu ,\nu )\in (\Lambda ({\mathfrak {g}}))^3\;:\; \lambda +\mu +\nu \in \Lambda _R({\mathfrak {g}})\}, \end{equation*}

where $\Lambda _R({\mathfrak {g}})$ is the root lattice of $\mathfrak {g}$ .

We now interpret ${\mathfrak {g}}\operatorname {-semigroup}$ in terms of § 2.2. Consider the semisimple, simply connected algebraic group $G$ with Lie algebra $\mathfrak {g}$ . Denote by $B$ and $T$ the connected subgroups of $G$ with Lie algebras $\mathfrak b$ and $\mathfrak t$ , respectively. The character groups $X(B)=X(T)=\Lambda ({\mathfrak {g}})$ of $B$ and $T$ coincide. For $\lambda \in X(T)$ , ${\mathcal L}_\lambda$ is the unique $G$ -linearized line bundle on the flag variety $G/B$ such that $B$ acts on the fiber over $B/B$ with weight $-\lambda$ .

Assume $X=(G/B)^3$ . Then ${\operatorname {Pic}}^G(X)$ identifies with $X(T)^3$ . For $(\lambda ,\mu ,\nu )\in X(T)^3$ , define ${\mathcal L}_{(\lambda ,\mu ,\nu )}$ . By the Borel–Weil theorem,

(10) \begin{align} {\operatorname {H}}^0(X,{\mathcal L}_{(\lambda ,\mu ,\nu )})=V(\lambda )^*\otimes V(\mu )^*\otimes V(\nu )^*. \end{align}

In particular, ${\operatorname {GIT-semigroup}}(G,X)\simeq {{\mathfrak {g}}\operatorname {-semigroup}}$ .

Given three parabolic subgroups $P,Q$ and $R$ containing $B$ , we consider more generally $X=G/P\times G/Q\times G/R$ . Then ${\operatorname {Pic}}^G(X)$ identifies with $X(P)\times X(Q)\times X(R)$ which is a subgroup of $X(T)^3$ . Moreover,

\begin{equation*} {\operatorname {GIT-semigroup}}(G,X)={\operatorname {GIT-semigroup}}(G,(G/B)^3)\cap (X(P)\times X(Q)\times X(R)). \end{equation*}

2.4 Schubert calculus

We need notation for the cohomology ring ${\operatorname {H}}^*(G/P,{{\mathbb {Z}}})$ ; $P\supset B$ being a parabolic subgroup. Let $W$ (respectively, $W_P$ ) be the Weyl group of $G$ (respectively, $P$ ). Let $\ell \,:\,W\longrightarrow {\mathbb {N}}$ be the Coxeter length, defined with respect to the simple reflections determined by the choice of $B$ . Let $W^P$ be the minimal length representatives of the cosets in $W/W_P$ .

For a closed irreducible subvariety $Z\subset G/P$ , let $[Z]$ be its class in ${\operatorname {H}}^*(G/P,{{\mathbb {Z}}})$ , of degree $2(\dim (G/P)-\dim (Z))$ . For $v \in W^P$ , set

\begin{equation*}\sigma _v=[\overline {BvP/P}]\end{equation*}

( $\dim (\overline {BvP/P})=\ell (v)$ ). Then

\begin{equation*} {\operatorname {H}}^*(G/P,{{\mathbb {Z}}})=\bigoplus _{v\in W^P}{{\mathbb {Z}}}\sigma _v. \end{equation*}

Let $w_0$ be the longest element of $W$ and $w_{0,P}$ be the longest element of $W_P$ . Set $v^\vee =w_0vw_{0,P}$ and $\sigma _v=\sigma ^{v^\vee }$ ; $\sigma ^v$ and $\sigma _v$ are Poincaré dual.

Let $\rho$ be the half sum of the positive roots of $G$ . To any one-parameter subgroup $\tau :{{\mathbb {C}}}^*\to T$ , associate the parabolic subgroup (see [Reference Mumford, Fogarty and KirwanMFK94])

\begin{equation*} P(\tau )=\Big\{g\in G\,:\,\lim _{t\to 0}\tau (t)g\tau (t{^{-1}}) \mbox { exists}\Big\}. \end{equation*}

Fix such a $\tau$ such that $P=P(\tau )$ .

For $v\in W^P$ , define the BK-degree of $\sigma ^v\in {\operatorname {H}}^*(G/P,{{\mathbb {Z}}})$ to be

\begin{equation*} {\operatorname {BK-deg}}(\sigma ^v):=\langle v{^{-1}}(\rho )-\rho ,\tau \rangle .\end{equation*}

Let $v_1,v_2$ and $v_3$ in $W^P$ . By [Reference Belkale and KumarBK06, Proposition 17], if $\sigma ^{v_3}$ appears in the product $\sigma ^{v_1} \cdot \sigma ^{v_2}$ then

(11) \begin{align} {\operatorname {BK-deg}}(\sigma ^{v_3})\leqslant {\operatorname {BK-deg}}(\sigma ^{v_1})+{\operatorname {BK-deg}}(\sigma ^{v_2}). \end{align}

In other words, the BK-degree filters the cohomology ring. Let $\odot _0$ denote the associated graded product on ${\operatorname {H}}^*(G/P,{{\mathbb {Z}}})$ .

2.5 Well-covering pairs

In [Reference RessayreRes10], ${\operatorname {GIT-sat}}(G,X)$ is described in terms of well-covering pairs. When $X=(G/B)^3$ , it recovers the description made by Belkale and Kumar [Reference Belkale and KumarBK06]. We now discuss the case when $X=G/P\times G/Q\times G/R$ is the product of three partial flag varieties of $G$ .

Let $\tau$ be a dominant one-parameter subgroup of $T$ . The centralizer $G^\tau$ of the image of $\tau$ in $G$ is a Levi subgroup. Moreover, $ P(\tau )$ is the parabolic subgroup generated by $B$ and $G^\tau$ . Let $C$ be an irreducible component of the fixed set $X^\tau$ of $\tau$ in $X$ . It is well known that $C$ is the $(G^\tau )^3$ -orbit of some $T$ -fixed point:

(12) \begin{align} C=G^\tau u{^{-1}} P/P\times G^\tau v{^{-1}} Q/Q \times G^\tau w{^{-1}} R/R, \end{align}

with $u\in {W_P}\backslash W/W_{P(\tau )}$ , and similarly for $v$ and $w$ . Set

\begin{equation*} C^+=P(\tau ) u{^{-1}} P/P\times P(\tau ) v{^{-1}} Q/Q \times P(\tau ) w{^{-1}} R/R. \end{equation*}

Then the closure of $C^+$ is a Schubert variety (for $G^3$ ) in $X$ . By [Reference RessayreRes10, Proposition 11], the pair $(C,\tau )$ is well covering if and only if

(13) \begin{align} [\overline {PuP(\tau )/P(\tau )}]\odot _0 [\overline {QvP(\tau )/P(\tau )}]\odot _0 [\overline {RwP(\tau )/P(\tau )}]=[pt] \in {\operatorname {H}}^*(G/P(\tau ),{{\mathbb {Z}}}). \end{align}

It is said to be dominant if

(14) \begin{align} [\overline {PuP(\tau )/P(\tau )}]\cdot [\overline {QvP(\tau )/P(\tau )}]\cdot [\overline {RwP(\tau )/P(\tau )}]\neq 0 \in {\operatorname {H}}^*(G/P(\tau ),{{\mathbb {Z}}}). \end{align}

In this paper, the reader can take these characterizations as definitions of well-covering and dominant pairs. They are used in [Reference RessayreRes10] to produce inequalities for the GIT-cones.

Proposition 2.1. Let $(\lambda ,\mu ,\nu )\in X(P)\times X(Q)\times X(R)$ be dominant, and let $\langle \cdot ,\cdot \rangle$ be the pairing between one parameter subgroups and characters of $T$ . Then the following are equivalent:

  1. (1) ${\mathcal L}_{(\lambda ,\mu ,\nu )}\in {\operatorname {GIT-sat}}(G,X)$ ;

  2. (2) $\langle u\tau ,\lambda \rangle +\langle v\tau ,\mu \rangle +\langle w\tau ,\nu \rangle \leqslant 0$ for all dominant pairs $(C,\tau )$ ;

  3. (3) $\langle u\tau ,\lambda \rangle +\langle v\tau ,\mu \rangle +\langle w\tau ,\nu \rangle \leqslant 0$ for all well-covering pairs $(C,\tau )$ .

Remark 2.2. By the definition of $\odot$ , if $(C,\tau )$ is well covering; it is also dominant. Hence, the inequalities in (3) are a subset of inequalities in (1).

In the case $P=Q=R=B$ , there is a more precise statement. The fact that the inequalites define the cone is due to Belkale and Kumar [Reference Belkale and KumarBK06, Theorem 28]. The irredundancy is [Reference RessayreRes10, Theorem B]. Let $\alpha$ be a simple root of $G$ . Denote by $P^\alpha$ the associated maximal parabolic subgroup of $G$ containing $B$ . Denote by $\varpi _{\alpha ^\vee }$ the associated fundamental one-parameter subgroup of $T$ characterized by $\langle \varpi _{\alpha ^\vee },\beta \rangle =\delta _{\alpha \beta }$ (Kronecker delta) for any simple root $\beta$ .

Theorem 2.3 ([Reference Belkale and KumarBK06, Theorem 28], [Reference RessayreRes10, Theorem B]). Here $X=(G/B)^3$ . Let $(\lambda ,\mu ,\nu )\in X(T)^3$ be dominant. Then, ${\mathcal L}_{(\lambda ,\mu ,\nu )}\in {\operatorname {GIT-sat}}(G,X)$ if and only if for any simple root $\alpha$ , for any $u,v,w$ in $W^{P^\alpha }$ such that

(15) \begin{align} [\overline {BuP^\alpha /P^\alpha }]\odot _0 [\overline {BvP^\alpha /P^\alpha }]\odot _0 [\overline {BwP^\alpha /P^\alpha }]=[pt] \in {\operatorname {H}}^*(G/P^\alpha ,{{\mathbb {Z}}}), \\[-30pt] \nonumber \end{align}
(16) \begin{align} \langle u\varpi _{\alpha ^\vee },\lambda \rangle +\langle v\varpi _{\alpha ^\vee },\mu \rangle +\langle w \varpi _{\alpha ^\vee },\nu \rangle \leqslant 0. \\[-8pt] \nonumber \end{align}

Moreover, this list of inequalities is irredundant.

Theorem 2.3 can be obtained from Proposition 2.1(3) by showing that it is sufficient to consider the one-parameter subgroups $\tau$ equal to $\varpi _{\alpha ^\vee }$ for some simple root $\alpha$ . See the proof of Theorem 5.1 below for a similar argument.

2.6 The eigencone

A relationship between ${\mathfrak {g}}\operatorname {-sat}$ and projections of coadjoint orbits was discovered by Heckman [Reference HeckmanHec82]. Theorem 2.4 below interprets ${\mathfrak {g}}\operatorname {-sat}$ in terms of eigenvalues.

Fix a maximal compact subgroup $U$ of $G$ such that $T\cap U$ is a Cartan subgroup of $U$ . Let $\mathfrak u$ and $\mathfrak t$ denote the Lie algebras of $U$ and $T$ , respectively. Let ${\mathfrak t}^+$ be the Weyl chamber of $\mathfrak t$ corresponding to $B$ . Let $\sqrt {-1}$ denote the usual complex number. It is well known that $\sqrt {-1}{\mathfrak t}^+$ is contained in $\mathfrak u$ and that the map

(17) \begin{align} \begin{array}{c@{\quad}c@{\quad}c} {\mathfrak t}^+&\longrightarrow &{\mathfrak u}/U\\ \xi &\longmapsto &U.(\sqrt {-1}\xi ) \end{array} \end{align}

is a homeomorphism. Here $U$ acts on $\mathfrak u$ by the adjoint action. Consider the set

\begin{equation*} \Gamma (U):=\{(\xi ,\zeta ,\eta )\in ({\mathfrak t}^+)^3\,:\, U.(\sqrt {-1}\xi )+U.(\sqrt {-1}\zeta )+U.(\sqrt {-1}\eta )\ni 0\}. \end{equation*}

Let ${\mathfrak u}^*$ (respectively, ${\mathfrak t}^*$ ) denote the dual (respectively, complex dual) of $\mathfrak u$ (respectively, $\mathfrak t$ ). Let ${\mathfrak t}^{*+}$ denote the dominant chamber of ${\mathfrak t}^*$ corresponding to $B$ . By taking the tangent map at the identity, one can embed $X(T)^+$ in ${\mathfrak t}^{*+}$ . Note that this embedding induces a rational structure on the complex vector space ${\mathfrak t}^*$ . Moreover, it allows the tensor cone ${\mathfrak {g}}\operatorname {-sat}$ to be embedded in $({\mathfrak t}^{*+})^3$ .

The Cartan-Killing form allows ${\mathfrak t}^+$ and ${\mathfrak t}^{*+}$ to be identified. In particular, $\Gamma (U)$ also embeds in $({\mathfrak t}^{*+})^3$ ; the subset of $({\mathfrak t}^{*+})^3$ thus obtained is denoted by $\tilde \Gamma (U)$ to avoid any confusion. The following result is well known; see, for example, [Reference KumarKum14, Theorem 5] and the references therein.

Theorem 2.4. The set $\Gamma (U)$ is a closed convex polyhedral cone. Moreover, ${\mathfrak {g}}\operatorname {-sat}$ is the set of the rational points in $\tilde \Gamma (U)$ .

3. The case of the symplectic group

3.1 The root system of type $C$

Let $V={{\mathbb {C}}}^{2n}$ with the standard basis $(\vec e_1,\vec e_2,\ldots ,\vec e_{2n})$ . Let $J_n$ be the $n\times n$ `anti-diagonal’ identity matrix and define a skew-symmetric bilinear form $\omega (\bullet ,\bullet ): V\times V\to {{\mathbb {C}}}$ using the block matrix $\Omega :=\left [\begin{matrix} 0 & J_n \\ -J_n & 0 \end{matrix}\right ]$ . By definition, the symplectic group $G={\operatorname {Sp}}(2n,{\mathbb {C}})$ is the group of automorphisms of $V$ that preserve this bilinear form.

Given an $n\times n$ matrix $A=(A_{ij})_{1\leqslant i,j\leqslant n}$ , define ${}^TA$ by $({}^TA)=A_{n+1-j\,n+1-i}$ , obtained from $A$ by reflection across the antidiagonal. The Lie algebra ${\mathfrak {sp}}(2n,{\mathbb {C}})$ is the set of matrices $M\in \textsf {Mat}_{2n\times 2n}({\mathbb {C}})$ such that ${}^tM\Omega +\Omega M=0$ ; namely,

(18) \begin{align} {\mathfrak {sp}}(2n,{\mathbb {C}})= \left \{ \begin{pmatrix} A&B\\ C&-{}^TA \end{pmatrix} : \begin{array}{l} A,B,C \mbox { of size }n\times n,\\ {}^TB=B \mbox { and } {}^TC=C \end{array} \right \} \end{align}

which has the complex dimension $2n^2+n$ . The Lie algebra ${\mathfrak u}(2n,{\mathbb {C}})$ of the unitary group $U(2n,{\mathbb {C}})$ is the set of anti-Hermitian matrices. Thus, (18) gives

(19) \begin{align} {\mathfrak {sp}}(2n,{\mathbb {C}})\cap {\mathfrak u}(2n,{\mathbb {C}})= \left \{ \begin{pmatrix} A&B\\ -{}^t\bar B&-{}^TA \end{pmatrix} : {}^t\bar A=-A \mbox { and } {}^TB=B \right \}, \end{align}

which has real dimension $2n^2+n$ . As a consequence, $U(2n)\cap {\operatorname {Sp}}(2n,{\mathbb {C}})$ is a maximal compact subgroup of ${\operatorname {Sp}}(2n,{\mathbb {C}})$ .

Let $B$ be the Borel subgroup of upper triangular matrices in $G$ . Let

\begin{equation*}T=\{{\operatorname {diag}}(t_1,\ldots ,t_{n},t_{n}^{-1},\ldots ,t_1^{-1})\,:\,t_i\in {\mathbb {C}}^*\}\end{equation*}

be the maximal torus contained in $B$ . For $i\in [n]$ , let $\varepsilon _i$ denote the character of $T$ that maps ${\operatorname {diag}}(t_1,\ldots ,t_{n},t_{n}^{-1},\ldots ,t_1^{-1})$ to $t_i$ ; then $X(T)=\oplus _{i=1}^n{{\mathbb {Z}}}\varepsilon _i$ . Here

\begin{equation*} \begin{array}{c} \Phi ^+=\{\varepsilon _i\pm \varepsilon _j\,:\,1\leqslant i\lt j\leqslant n\}\cup \{2\varepsilon _i\,:\,1\leqslant i\leqslant n\}, \\[6pt] \Delta =\{\alpha _1=\varepsilon _1-\varepsilon _2,\,\alpha _2=\varepsilon _2-\varepsilon _3,\ldots ,\, \alpha _{n-1}=\varepsilon _{n-1}-\varepsilon _{n},\,\alpha _{n}=2\varepsilon _{n}\},\\[6pt] X(T)^+=\displaystyle\left\{\sum \nolimits_{i=1}^{n}\lambda _i \varepsilon _i\,:\, \lambda _1\geqslant \cdots \geqslant \lambda _{n}\geqslant 0\right\}={\operatorname {Par}}_n. \end{array} \end{equation*}

For $i\in [2n]$ , set $\overline {i}=2n+1-i$ . The Weyl group $W$ of $G$ may be identified with a subgroup of the Weyl group $S_{2n}$ of ${\operatorname {SL}}(V)$ . More precisely,

\begin{equation*} W=\{w\in S_{2n}\,:\,w(\overline {i})=\overline {w(i)} \ \ \forall i\in [2n]\}. \end{equation*}

Observe that $T\cap U(2n,{\mathbb {C}})$ has real dimension $n$ and is a maximal torus of $U(2n)\cap {\operatorname {Sp}}(2n,{\mathbb {C}})$ . The bijection (17) implies that any matrix $M_1=\begin{pmatrix} A&B\\ -{}^t\bar B&-{}^TA \end{pmatrix}$ in ${\mathfrak {sp}}(2n,{\mathbb {C}})\cap {\mathfrak u}(2n,{\mathbb {C}})$ (see (19)) is diagonalizable with eigenvalues in $\sqrt {-1}{{\mathbb {R}}}$ . Moreover, with the eigenvalues in nonincreasing order, we get

\begin{equation*} \lambda (\sqrt {-1}M_1)\in \{(\lambda _1\geqslant \cdots \geqslant \lambda _n\geqslant -\lambda _n\geqslant \cdots \geqslant -\lambda _1)\,:\,\lambda _i\in {{\mathbb {R}}}\}. \end{equation*}

For $\lambda =(\lambda _1,\dots ,\lambda _n)\in {\operatorname {Par}}_n$ , set $\hat \lambda =(\lambda _1,\dots ,\lambda _n,-\lambda _n,\dots ,-\lambda _1)$ . Now, Theorems 1.1 with $m=n$ and Theorem 2.4 give an interpretation of ${\operatorname {NL-sat}}(n)$ in terms of eigenvalues:

Proposition 3.1. Let $\lambda ,\mu ,\nu \in {\operatorname {Par}}_n$ . Then $(\lambda ,\mu ,\nu )\in {\operatorname {NL-sat}}(n)$ if and only if there exist three matrices $M_1,M_2,M_3\in {\mathfrak {sp}}(2n,{\mathbb {C}})\cap {\mathfrak u}(2n,{\mathbb {C}})$ such that $M_1+M_2+M_3=0$ and

\begin{equation*} (\hat \lambda ,\hat \mu ,\hat \nu )=(\lambda (\sqrt {-1}M_1), \lambda (\sqrt {-1}M_2), \lambda (\sqrt {-1}M_3)). \end{equation*}

3.2 Isotropic Grassmannians and Schubert classes

Our reference for this subsection is [Reference RessayreRes12, § 5]. For $r=1,\dots ,n$ , the one-parameter subgroup $\varpi _{\alpha _r^\vee }$ is given by

\begin{equation*}\varpi _{\alpha _r^\vee } (t)={\operatorname {diag}}(t,\dots ,t,1,\dots ,1,t{^{-1}},\dots ,t{^{-1}}),\end{equation*}

where $t$ and $t{^{-1}}$ occur $r$ times.

A subspace $W\subseteq V$ is isotropic if for all $\vec v, \vec v^{\prime}\in W$ , $\omega (\vec v, \vec v^{\prime})=0$ . Given an $r$ -subset $I\subset [2n]$ , we set $F_I={\operatorname {Span}}(\vec e_i:i\in I)$ . Clearly, $F_I$ is isotropic if and only if $I\cap \bar I=\emptyset$ , where $\bar I=\{\bar i\,:\,i\in I\}$ . Now, $P^{\alpha _r}$ is the stabilizer of the isotropic subspace $F_{\{1,\dots ,r\}}$ . Thus, $G/P^{\alpha _r}={\operatorname {Gr}}_\omega (r,2n)$ is the Grassmannian of isotropic $r$ -dimensional vector subspaces of $V$ .

Let ${\mathcal S}(r,N)$ denote the set of subsets of $\{1,\dots ,N\}$ with $r$ elements. Set

\begin{equation*} \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n)):=\{I\in {\mathcal S}(r,2n)\,:\, I\cap \overline {I}=\emptyset \}. \end{equation*}

If $I=\{i_1\lt \cdots \lt i_r\}\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))$ , let $i_{\overline {k}}:=\overline {i_k}$ for $k\in [r]$ , and $\{i_{r+1}\lt \cdots \lt i_{\overline {r+1}}\}=[2n]-(I\cup \overline {I})$ . Therefore, $w_I=(i_1,\dots ,i_{2n})\in S_{2n}$ is the element of $W^{P^{\alpha _r}}$ corresponding to $F_I$ ; that is, $F_I=w_I P^{\alpha _r}/P^{\alpha _r}$ .

Set

\begin{equation*} \operatorname {Schub}^{\prime}({\operatorname {Gr}}_\omega (r,2n)):= \left \{ \begin{array}{l@{\quad}l} (A,A^{\prime})\,:\, &A\in {\mathcal S}(a,n),\, A^{\prime}\in {\mathcal S}(a^{\prime},n)\mbox { for some $a$ and $a^{\prime}$}\\ &\mbox{s.t.}\;\;a+a^{\prime}=r\mbox { and } A\cap A^{\prime}=\emptyset \end{array} \right \}. \end{equation*}

This map is a bijection:

(20) \begin{align} \begin{array}{c@{\quad}c@{\quad}l} \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))&\longrightarrow &\operatorname {Schub}^{\prime}({\operatorname {Gr}}_\omega (r,2n))\\ I&\longmapsto &(\bar I\cap [n],I\cap [n]). \end{array} \end{align}

Recall from the introduction the definition of $\tau (I)$ and hence $\tau (A)$ and $\tau (A^{\prime})$ . The relationship between these three partitions is depicted in Figure 1. In particular, note that $\tau (A^{\prime})\subseteq \tau (I)$ .

Given $I\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))$ for some $1\leqslant r\leqslant n$ , set $I^2\in {\mathcal S}(r,2r)$ and $I^0\in {\mathcal S}(r,2n-r)$ to be the unique $r$ -element sets, such that

\begin{equation*}\tau (I^2) = \tau (I,I\cup \bar {I})\text { and }\tau (I^0) = \tau (I,[2n]- \bar {I}).\end{equation*}

Figure 1. $\tau (I)$ , $\tau (A)$ and $\tau (A^{\prime})\; (\tau (A^{\prime})\subseteq \tau (I))$ .

Example 3.2. Let $I = \{1,3,5\} \in \operatorname {Schub}({\operatorname {Gr}}_\omega (3,8))$ . Then $w_I = 13527468\in S_8$ ,

\begin{equation*} \tau (I^2) = \tau (\{1,3,5\},\{1,3,4,5,6,8\}) = 100 \text { and } \tau (I^0) = \tau (\{1,3,5\},\{1,2,3,5,7\}) = 110. \end{equation*}

Thus $I^2 = \{1,2,4\},I^0 = \{1,3,4\}$ . By Equation (20), $A = \bar {I}\cap [n] = \{4\}$ and $A^{\prime} = I\cap [n] = \{1,3\}$ .

Definition 3.3. Fix a partition $\lambda = (\lambda _1,\ldots ,\lambda _k) \subseteq (a^b)$ , that is, the rectangle with $a$ columns and $b$ rows. Define $\lambda ^\vee$ with respect to $(a^b)$ to be the partition $(a-\lambda _b,a-\lambda _{b-1},\ldots ,a-\lambda _1)$ where we set $\lambda _i = 0$ for $i\gt k$ . We will denote this by $\lambda ^{\vee [a^b]}$ .

Now, by [Reference Belkale and KumarBK10, Proposition 32],

(21) \begin{align} {\operatorname {codim}}(\overline {BF_I})= |\tau (I^0)^{\vee [(2n-2r)^r]}| + 1/2(|\tau (I^2)^{\vee [r^r]}| + |I\cap [n]|). \end{align}

Moreover,

(22) \begin{align} \dim ({\operatorname {Gr}}_\omega (r,2n))= r(2n-2r)+\frac {r(r+1)}{2}. \end{align}

Let $I\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))$ and $A=\bar I\cap [n]$ , $A^{\prime}=I\cap [n]$ be the corresponding pair in $\operatorname {Schub}^{\prime}({\operatorname {Gr}}_\omega (r,2n))$ . Set

(23) \begin{align} \tau ^0(A,A^{\prime})=\tau (I^0),\quad \tau ^2(A,A^{\prime})=\tau (I^2). \end{align}

For later use, observe that

(24) \begin{align} \tau (I)=\tau (I^0)+\tau (I^2). \end{align}

While the above discussion defines $\tau ^0(A,A^{\prime}), \tau ^2(A,A^{\prime})$ through the bijection (20), we emphasize that these partitions from Theorem 1.2 can be defined explicitly.

Definition-Lemma 3.4. Set $a=|A|$ and $a^{\prime}=|A^{\prime}|$ . Write $A=\{\alpha _1\lt \cdots \lt \alpha _a\}$ and $A^{\prime}=\{\alpha ^{\prime}_1\lt \cdots \lt \alpha ^{\prime}_{a^{\prime}}\}$ . Then

\begin{equation*} \begin{array}{rl@{\quad}rl} \tau ^2(A,A^{\prime})_k&=a+|A^{\prime}\cap [\alpha _k,n]|&\forall k&=1,\dots ,a;\\[4pt] \tau ^2(A,A^{\prime})_{l+a}&=|A\cap [\alpha ^{\prime}_{a^{\prime}+1-l}]|&\forall l&=1,\dots ,a^{\prime};\\[4pt] \tau ^0(A,A^{\prime})_k&=n-a-a^{\prime}+|[\alpha _k,n] - (A\cup A^{\prime})| &\forall k&=1,\dots ,a;\\[4pt] \tau ^0(A,A^{\prime})_{l+a}&=|[\alpha ^{\prime}_{a^{\prime}+1-l}]- (A\cup A^{\prime})|&\forall l&=1,\dots ,a^{\prime}. \end{array} \end{equation*}

Proof. Write $I=A^{\prime}\cup \bar A=\{i_1\lt \cdots \lt i_r\}$ with $r=a+a^{\prime}$ . By definition,

\begin{equation*}\tau ^2(A,A^{\prime})_k:=\tau (I^2)_k=|\bar I\cap [i_{a+a^{\prime}+1-k}]|\quad \text {for $1\leqslant k\leqslant a+a^{\prime}$}.\end{equation*}

If $k\leqslant a$ , then $i_{a+a^{\prime}+1-k}\in \bar A\subset [n+1,2n]$ , $i_{a+a^{\prime}+1-k}=\overline {\alpha _k}$ and $\bar I\cap [i_{a+a^{\prime}+1-k}]=A\cup \overline {A^{\prime}\cap [\alpha _k,n]}$ . The first assertion follows.

If $k=a+l$ for some positive $l$ , then $i_{a+a^{\prime}+1-k}\in A^{\prime}\subseteq [n]$ , $i_{a+a^{\prime}+1-k}=\alpha ^{\prime}_{a^{\prime}+1-l}$ and $\bar I\cap [i_{a+a^{\prime}+1-k}]=A\cap [\alpha ^{\prime}_{a^{\prime}+1-l}]$ .

Similarly,

\begin{equation*} \tau ^0(A,A^{\prime})_k:=\tau (I^0)_k=|[i_{a+a^{\prime}+1-k}]\cap ([2n]-(I\cup \bar I)| \quad\text{for $1\leqslant k\leqslant a+a^{\prime}$.} \end{equation*}

If $k\leqslant a$ , then $[i_{a+a^{\prime}+1-k}]\cap ([2n]-(I\cup \bar I))= ([n]-(A\cup A^{\prime}))\cup \overline {[\alpha _k,n]-(A\cup A^{\prime})}$ (a disjoint union). This proves the third claim.

If $k=a+l$ with some positive $l$ , then $\alpha ^{\prime}_{a^{\prime}+1-l}=i_{a+a^{\prime}+1-k}\in [n]$ ; the last assertion follows.

3.3 The parabolic subgroup $P_0$

Fix $m\geqslant n$ . Let $P_0$ be the subgroup of ${\operatorname {Sp}}(2m,{\mathbb {C}})$ of matrices

(25) \begin{align} \left(\begin{array}{c@{\quad}c@{\quad}c} T_1&*&*\\ 0&A&*\\ 0&0&T_2 \end{array} \right), \end{align}

where $T_1$ and $T_2$ are $n\times n$ upper-triangular matrices and $A$ is a matrix in ${\operatorname {Sp}}(2m-2n,{\mathbb {C}})$ . $P_0$ is the standard parabolic subgroup of ${\operatorname {Sp}}(2m,{\mathbb {C}})$ corresponding to the simple roots $\{\alpha _{n+1},\dots ,\alpha _{m}\}$ . A character $\lambda =\sum _{i=1}^m\lambda _i\varepsilon _i\in X(T)$ extends to $P_0$ if and only if $\lambda _{n+1}=\cdots =\lambda _m=0$ . Thus the set of dominant characters of $X(P_0)$ identifies with ${\operatorname {Par}}_n$ . Hence,

(26) \begin{align} {{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3={\operatorname {GIT-sat}}({\operatorname {Sp}}(2m,{\mathbb {C}}),({\operatorname {Sp}}(2m,{\mathbb {C}})/P_0)^3). \end{align}

Let $\operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (r,2m))$ be the set of $I\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2m))$ such that the Schubert variety $\overline {BF_I}$ is $P_0$ -stable. Then $I\in \operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (r,2m))$ if and only if $w_I\in W^{P^{\alpha _r}}$ and $s_{\alpha _i}w_I\leqslant w_I$ (cover in Bruhat order for $W$ ) for all $i\in [n+1,m]$ . Since the simple transposition $s_{\alpha _i}$ swaps $i$ and $\bar {i}$ with $i+i$ and $\bar {i+1}$ respectively if $i\lt m$ , and swaps $m$ with $\bar m$ if $i = m$ , we have

(27) \begin{align} I\in \operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (r,2m)) \iff I\cap [n+1,2m-n]=[k,2m-n] \,\text{for some $k\geqslant m+1$.} \end{align}

4. Proof of Theorems 1.1 and 1.2

Proposition 4.1. The inequalities (4) in Theorem 1.2 characterize ${{\mathfrak {sp}}\operatorname {-sat}}(n)$ .

Proof. Since ${{\mathfrak {sp}}\operatorname {-sat}}(n)={\operatorname {GIT-sat}}({\operatorname {Sp}}(2n), ({\operatorname {Sp}}(2n)/B)^3$ (see § 2.3), we may apply Theorem 2.3. Let $(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3$ . Write $\lambda =\sum _i\lambda _i\varepsilon _i$ , and similarly for $\mu$ and $\nu$ .

Fix $1\leqslant r\leqslant n$ and $\alpha =\alpha _r\in \Delta$ . Given $I\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))$ , from the description of $\varpi _{\alpha _r^{\vee }}$ and $w_I$ it is easy to check that

(28) \begin{align} \langle w_I\varpi _{\alpha _r^\vee },\lambda \rangle =\sum _{i\in I\cap [n]}\lambda _i-\sum _{i\in \bar I\cap [n]}\lambda _i \end{align}

Then (4) is obtained from (16) associated to the triple of Schubert classes $(I,J,K)\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))^3$ by setting

\begin{align*} A &= \bar {I}\cap [n],\ \ \; A^{\prime} = I\cap [n],\\ B &= \bar {J}\cap [n],\ \ B^{\prime} = J\cap [n],\\C &= \bar {K}\cap [n],\ C^{\prime} = K\cap [n]. \end{align*}

Since the map (20) is bijective, it suffices to show (15) from Theorem 2.3 is equivalent to:

  1. (1) $|A^{\prime}|+|B^{\prime}|+|C^{\prime}|=r$ ; and

  2. (2) $c_{\tau ^0(A,A^{\prime})^{\vee [(2n-2r)^r]}, \tau ^0(B,B^{\prime})^{\vee [(2n-2r)^r]}}^{\tau ^0(C,C^{\prime})}= c_{\tau ^2(A,A^{\prime})^{\vee [r^r]}, \tau ^2(B,B^{\prime})^{\vee [r^r]}}^{\tau ^2(C,C^{\prime})}=1$ .

By [Reference RessayreRes12, Theorem 19], condition (15) is equivalent to:

  1. (1) ${\operatorname {codim}}(\overline {BF_I})+{\operatorname {codim}}(\overline {BF_J})+{\operatorname {codim}}(\overline {BF_K})=\dim ({\operatorname {Gr}}_\omega (r,2n))$ ; and

  2. (2) $c^{\tau (K^0)}_{\tau (I^0)^{\vee [(2n-2r)^r]}, \tau (J^0)^{\vee [(2n-2r)^r]}} = c^{\tau (K^2)}_{\tau (I^2)^{\vee [r^r]}, \tau (J^2)^{\vee [r^r]}} = 1$ .

By definition, the two conditions involving Littlewood–Richardson are the same. Assuming these two Littlewood–Richardson coefficients equal to one 1, it remains to prove that ${\operatorname {codim}}(\overline {BF_I})+{\operatorname {codim}}(\overline {BF_J})+{\operatorname {codim}}(\overline {BF_K})=\dim ({\operatorname {Gr}}_\omega (r,2n))$ if and only if $|A^{\prime}|+|B^{\prime}|+|C^{\prime}|=r$ . This directly follows from (21) and (22).

Proof of Theorem 1.1. By Theorem 2.4, the inclusion ${{\mathfrak {sp}}\operatorname {-sat}}(n)\subset {{\mathfrak {sp}}\operatorname {-sat}}(m)$ is equivalent to the inclusion $\Gamma ({\operatorname {Sp}}(2n,{\mathbb {C}})\cap U(2n,{\mathbb {C}}))\subset \Gamma ({\operatorname {Sp}}(2m,{\mathbb {C}})\cap U(2m,{\mathbb {C}}))$ . Here we use the symplectic form defined in § 3.1 to embed ${\operatorname {Sp}}(2n,{\mathbb {C}})$ in ${\operatorname {GL}}(2n,{\mathbb {C}})$ .

Clearly, the following map is well defined:

\begin{equation*} \begin{array}{c@{\quad}c@{\quad}l} {\operatorname {Lie}}({\operatorname {Sp}}(2n,{\mathbb {C}})\cap U(2n,{\mathbb {C}}))&\longrightarrow &{\operatorname {Lie}}({\operatorname {Sp}}(2m,{\mathbb {C}})\cap U(2m,{\mathbb {C}}))\\[5pt]M=\left(\begin{array}{l@{\quad}l} A&B\\[3pt]C&D \end{array}\right) &\longmapsto & \tilde M= \left(\begin{array}{l@{\quad}l@{\quad}l} A&0&B\\[3pt]0&0&0\\[3pt]C&0&D \end{array}\right) \end{array} \end{equation*}

where $A$ , $B$ , $C$ and $D$ are square matrices of size $n$ , and the matrices of these Lie algebras are described by (19).

Let $(\hat h_1,\hat h_2,\hat h_3)\in \Gamma ({\operatorname {Sp}}(2n,{\mathbb {C}}) \cap U(2n,{\mathbb {C}}))$ . Let

\begin{equation*}(M_1,M_2,M_3)\in ({\operatorname {Sp}}(2n,{\mathbb {C}}) \cap U(2n,{\mathbb {C}}))^3.(\sqrt {-1}\hat h_1, \sqrt {-1}\hat h_2, \sqrt {-1}\hat h_3)\end{equation*}

such that $M_1+M_2+M_3=0$ .

The fact that $\tilde M_1+\tilde M_2+\tilde M_3=0$ implies that $(\hat h_1,\hat h_2,\hat h_3)\in \Gamma ({\operatorname {Sp}}(2m,{\mathbb {C}}) \cap U(2m,{\mathbb {C}}))$ , where $\hat h_1,\hat h_2,\hat h_3$ are viewed as elements of ${\operatorname {Par}}_{m}$ by postpending $m-n$ many $0$ s.

To obtain the converse inclusion

\begin{equation*}{\operatorname {GIT-sat}}({\operatorname {Sp}}(2m,{\mathbb {C}}),({\operatorname {Sp}}(2m,{\mathbb {C}})/P_0)^3)={{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3\subset {{\mathfrak {sp}}\operatorname {-sat}}(n),\end{equation*}

we have to prove that any inequality (4) from Proposition 4.1 is satisfied by the points of ${{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3$ ; here we have used (26). Fix such an inequality $(A,A^{\prime},B,B^{\prime},C,C^{\prime})$ . Set

\begin{equation*}I=A^{\prime}\cup \bar A\subset [2n],\quad J=B^{\prime}\cup \bar B\subset [2n], \quad K=C^{\prime}\cup \bar C\subset [2n].\end{equation*}

Similarly, for $m$ , set

\begin{equation*}\tilde I=A^{\prime}\cup \{2m+1-i\,:\,i\in A\},\quad \tilde J=B^{\prime}\cup \{2m+1-i\,:\,i\in B\}\end{equation*}

and

\begin{equation*}\tilde K=C^{\prime}\cup \{2m+1-i\,:\,i\in C\};\end{equation*}

these are subsets of $[2m]$ . Set also $a^{\prime}=|A^{\prime}|$ , $b^{\prime}=|B^{\prime}|$ and $c=|C|$ .

Notice that $(\tilde I^2)^0,(\tilde J^2)^0,(\tilde K^2)^0\subseteq 0^r = \emptyset$ . Thus, trivially,

(29) \begin{align} c_{\tau ((\tilde I^2)^0)^\vee , \tau ((\tilde J^2)^0)^\vee }^{\tau ((\tilde K^2)^0)}=c_{\emptyset ,\emptyset }^{\emptyset }=1. \end{align}

Also, $(\tilde I^2)^2=\tilde I^2, (\tilde J^2)^2=\tilde J^2, (\tilde K^2)^2=\tilde K^2$ . Since $\tau (\tilde I^2)=\tau (I^2):=\tau ^2(A,A^{\prime})$ , $\tau (\tilde J^2)=\tau (J^2):=\tau ^2(B,B^{\prime})$ and $\tau (\tilde K^2)=\tau (K^2):=\tau ^2(C,C^{\prime})$ , we have

(30) \begin{align} c_{\tau ((\tilde I^2)^2)^{\vee [r^r]}, \tau ((\tilde J^2)^2)^{\vee [r^r]}}^{\tau ((\tilde K^2)^2)}= c_{\tau (\tilde I^2)^{\vee [r^r]},\tau (\tilde J^2)^{\vee [r^r]}}^{\tau (\tilde K^2)}= c_{\tau ^2(A,A^{\prime})^{\vee [r^r]}\; \tau ^2(B,B^{\prime})^{\vee [r^r]}}^{\tau ^2(C,C^{\prime})}=1. \end{align}

We apply [Reference RessayreRes12, Theorem 8.2] to ${\operatorname {Gr}}_\omega (r,2r)$ and the triple $\tilde I^2, \tilde J^2, \tilde K^2$ . Equations (29) and (30) mean that condition (iii) of the said theorem holds. Hence by part (ii) of ibid.,

(31) \begin{align} [\overline {BF_{\tilde I^2}}]\cdot [\overline {BF_{\tilde J^2}}]\cdot [\overline {BF_{\tilde K^2}}]=[pt] \in {\operatorname {H}}^*({\operatorname {Gr}}_\omega (r,2r),{{\mathbb {Z}}}). \end{align}

One can easily check that

\begin{equation*} \begin{array}{c} \tau (\tilde K^0)=[2(m-n)]^{c}+\tau (K^0),\\ \tau (\tilde I^0)^{\vee [(2m-2r)^r]}=[2(m-n)]^{a^{\prime}}+\tau (I^0)^{\vee [(2n-2r)^r]},\\ \tau (\tilde J^0)^{\vee [(2m-2r)^r]}=[2(m-n)]^{b^{\prime}}+\tau (J^0)^{\vee [(2n-2r)^r]}. \end{array} \end{equation*}

The assumption $a^{\prime}+b^{\prime}=c$ and the semigroup property of $\operatorname {LR-semigroup}$ implies that

(32) \begin{align} c_{\tau (\tilde I^0)^{\vee [(2m-2r)^r]}, \tau (\tilde J^0)^{\vee [(2m-2r)^r]}}^{\tau (\tilde K^0)}\neq 0. \end{align}

Next we apply [Reference RessayreRes12, Proposition 8.1] to $\tilde I, \tilde J, \tilde K$ and the space ${\operatorname {Gr}}_\omega (r,2m)$ ; Equations (31) and (32) mean that condition (iii) holds. Hence by (i) of ibid. and (27),

(33) \begin{align} [\overline {P_0F_{\tilde I}}]\odot _0 [\overline {P_0F_{\tilde J}}]\odot _0 [\overline {P_0F_{\tilde K}}]=d[pt] \in {\operatorname {H}}^*({\operatorname {Gr}}_\omega (r,2m),{{\mathbb {Z}}}), \end{align}

for some nonzero $d$ . Now use Proposition 2.1, which shows that (4) is a case of Proposition 2.1(2) which holds on ${\operatorname {GIT-sat}}({\operatorname {Sp}}(2m,{\mathbb {C}}),({\operatorname {Sp}}(2m,{\mathbb {C}})/P_0)^3)={{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3$ , as desired.

Proof of Theorem 1.2. This follows from Theorem 1.1 and Proposition 4.1.

Example 4.2. Let $n=4, r=3$ . Let

\begin{equation*} A=B^{\prime}=C^{\prime}=\emptyset ,\quad A^{\prime}=\{2,3,4\},\quad B=\{1,2,4\},\quad C=\{1,3,4\}, \end{equation*}

giving a triple $((A,A^{\prime}),(B,B^{\prime}),(C,C^{\prime}))$ in $(\operatorname {Schub}^{\prime}({\operatorname {Gr}}_\omega (3,8)))^3$ satisfying conditions (1) and (2) from Theorem 1.2. The corresponding triple in $\operatorname {Schub}({\operatorname {Gr}}_\omega (3,8))$ is

\begin{equation*}I=\{2, 3, 4\},\quad J=\{5, 7, 8\},\quad K=\{5, 6, 8\} \subseteq [8].\end{equation*}

Thus

\begin{equation*}\tau (I)=(1,1,1),\quad \tau (J)=(5,5,4),\quad \tau (K)=(5,4,4)\subseteq (5^3).\end{equation*}

Now,

\begin{equation*} \begin{array}{c} \ \tau (I^0) = \tau (\{2,3,4\},\{1,2,3,4,8\}) = 111,\quad \ \ \tau (I^2) = \tau (\{2,3,4\},\{2,3,4,5,6,7\})=000,\\ {\kern-1pt}\ \tau (J^0) = \tau (\{5,7,8\},\{3,5,6,7,8\}) = 221,\quad\ {\kern2pt} \tau (J^2) = \tau (\{5,7,8\},\{1,2,4,5,7,8\}) = 333,\\ \tau (K^0) = \tau (\{5,6,8\},\{2,5,6,7,8\}) = 211,\quad\ \tau (K^2)=\tau (\{5,6,8\},\{1,3,4,5,6,8\})=333, \end{array} \end{equation*}

and thus

\begin{equation*} \begin{array}{c} I^0 = \{2,3,4\},\quad\ \ I^2 = \{1,2,3\},\\ {\kern-1pt}J^0= \{2,4,5\},\quad\ {\kern1.8pt} J^2 = \{4,5,6\},\\{\kern-3.3pt} K^0 = \{2,4,5\},\quad {\kern3.3pt}K^2=\{4,5,6\}. \end{array} \end{equation*}

The reader can check that

\begin{equation*} \begin{array}{c} c_{\tau ^0(A,A^{\prime})^{\vee [(2n-2r)^r]}, \tau ^0(B,B^{\prime})^{\vee [(2n-2r)^r]}}^{\tau ^0(C,C^{\prime})}=c^{\tau (K_0)}_{\tau (I_0)^{\vee [(2n-2r)^r]}, \tau (J_0)^{\vee [(2n-2r)^r]}} =c_{(1,1,1),(1)}^{(2,1,1)}=1,\\[10pt] c_{\tau ^2(A,A^{\prime})^{\vee [r^r]}, \tau ^2(B,B^{\prime})^{\vee [r^r]}}^{\tau ^2(C,C^{\prime})}=c^{\tau (K_2)}_{\tau (I_2)^{\vee [r^r]}, \tau (J_2)^{\vee [r^r]}} = c_{(3,3,3),(0,0,0)}^{(3,3,3)} =1. \end{array} \end{equation*}

Hence, by Theorem 1.2, $-\lambda _2-\lambda _3-\lambda _4+\mu _1+\mu _2+\mu _4+\nu _1+\nu _3+\nu _4\geqslant 0$ is one of the inequalities defining ${{\mathfrak {sp}}\operatorname {-sat}}(4)$ .

5. The truncated tensor cone

In this section, we characterize the truncated tensor cone of ${{\mathfrak {sp}}\operatorname {-sat}}(m)$ , that is, ${{\mathfrak {sp}}\operatorname {-sat}}(m)\cap ({\operatorname {Par}}_n^{\mathbb {Q}})^3$ where $m\gt n$ . By (3), this implies another set of inequalities for ${\operatorname {NL-sat}}(n)$ .

We first need the following result, a generalization of Theorem 2.3.

Theorem 5.1. Here $X=G/P\times G/Q\times G/R$ . Let $(\lambda ,\mu ,\nu )\in X(P)\times X(Q)\times X(R)$ be dominant. Then ${\mathcal L}_{(\lambda ,\mu ,\nu )}\in {\operatorname {GIT-sat}}(G,X)$ if and only if for any simple root $\alpha$ , for any

\begin{equation*} (u,v,w)\in {W_P}\backslash W/ W_{P^\alpha }\times {W_Q}\backslash W/W_{P^\alpha }\times {W_R}\backslash W/W_{P^\alpha } \end{equation*}

such that

(34) \begin{align} [\overline {PuP^\alpha /P^\alpha }]\odot _0 [\overline {QvP^\alpha /P^\alpha }]\odot _0 [\overline {RwP^\alpha /P^\alpha }]=[pt] \in {\operatorname {H}}^*(G/P^\alpha ,{{\mathbb {Z}}}), \\[-30pt] \nonumber \end{align}
(35) \begin{align} \langle u\varpi _{\alpha ^\vee },\lambda \rangle +\langle v\varpi _{\alpha ^\vee },\mu \rangle +\langle w\varpi _{\alpha ^\vee },\nu \rangle \leqslant 0. \\[-8pt] \nonumber \end{align}

Proof. ${\operatorname {GIT-sat}}(G,X)$ is characterized by Proposition 2.1(3); let $(C,\tau )$ and a choice of

\begin{equation*}(u^{\prime},v^{\prime},w^{\prime})\in {W_P}\backslash W/ W_{P(\tau )}\times {W_Q}\backslash W/W_{P(\tau )}\times {W_R}\backslash W/W_{P(\tau )}\end{equation*}

be as in that proposition. Since every inequality (35) appears in part (3) of Proposition 2.1 with $\tau = \varpi _{\alpha ^\vee }$ , it suffices to show that inequalities in (3) of Proposition 2.1 are implied by the inequalities in Theorem 5.1.

Write

\begin{equation*} \tau =\sum _{\alpha \in \Delta }n_\alpha \varpi _{\alpha ^\vee }, \end{equation*}

where $\Delta$ is the set of simple roots. Since $\tau$ is dominant the $n_\alpha$ are nonnegative. Set

\begin{equation*} {\operatorname {Supp}}(\tau ):=\{\alpha \in \Delta \,:\,n_\alpha \neq 0\}. \end{equation*}

Fix any $\alpha \in {\operatorname {Supp}}(\tau )$ ; $P^\alpha :=P(\varpi _{\alpha ^\vee })$ contains $P(\tau )$ . Let

\begin{equation*}\pi \,:\, G/P(\tau )\longrightarrow G/P^\alpha \end{equation*}

denote the associated projection. By [Reference RichmondRic12, Theorem 1.1 and § 1.1] (see also [Reference RessayreRes11]), condition (13) implies there are

\begin{equation*}(u,v,w)\in {W_P}\backslash W/ W_{P^\alpha }\times {W_Q}\backslash W/W_{P^\alpha }\times {W_R}\backslash W/W_{P^\alpha },\end{equation*}

such that condition (34) holds and such that $(u,v,w)$ and $(u^{\prime},v^{\prime},w^{\prime})$ define the same cosets in ${W_P}\backslash W/ W_{P^\alpha }\times {W_Q}\backslash W/W_{P^\alpha }\times {W_R}\backslash W/W_{P^\alpha }$ . Therefore, inequality (35) is the same as

\begin{equation*}\langle u^{\prime}\varpi _{\alpha ^\vee },\lambda \rangle +\langle v^{\prime}\varpi _{\alpha ^\vee },\mu \rangle +\langle w^{\prime}\varpi _{\alpha ^\vee },\nu \rangle \leqslant 0.\end{equation*}

Therefore each of the inequalities (3) can be written as a linear combination of (35). Hence, the inequalities of the theorem imply and are implied by the inequalities of Proposition 2.1 (3), so the result follows.

We now deduce from Theorem 5.1 the following statement.

Proposition 5.2. Let $(\lambda ,\mu ,\nu )$ in ${\operatorname {Par}}_n$ and $m\geqslant n$ . Then $(\lambda ,\mu ,\nu )\in {{\mathfrak {sp}}\operatorname {-sat}}(m)$ if and only if

(36) \begin{align} |\lambda _{I\cap [n]}|- |\lambda _{\bar I \cap [n]}|+|\mu _{J\cap [n]}| -|\mu _{\bar J\cap [n]}|+|\nu _{K\cap [n]}| -|\nu _{\bar K\cap [n]}|\leqslant 0, \end{align}

for any $1\leqslant r\leqslant m$ and $(I,J,K)\in \operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (r,2m))^3$ such that:

  1. (1) $|I\cap [m]|+|J\cap [m]|+|K\cap [m]| = r$ ; and

  2. (2) $ c_{\tau (I_0)^{\vee [(2m-2r)^r]},\tau (J_0)^{\vee [(2m-2r)^r]}}^{\tau (K_0)}=c_{\tau (I_2)^{\vee [r^r]},\tau (J_2)^{\vee [r^r]}}^{\tau (K_2)}=1.$

Proof. We already observed (28) that inequality (36) is inequality (3) in our context. Regarding Theorem 5.1, the only thing to prove is that condition (13) associated to $(I,J,K)$ is equivalent to the two conditions of the proposition. This is [Reference RessayreRes12, Theorem 8.2].

A priori, Proposition 5.2 could contain redundant inequalities. In view of Theorem 1.2, an affirmative answer to this question would imply irredundancy:

Question 1. Does any $(I,J,K)\in \operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (r,2m))^3$ occurring in Proposition 5.2 satisfy

  1. (1) $I\cap [n+1,2m-n]=J\cap [n+1,2m-n]=K\cap [n+1,2m-n]=\emptyset$ ,

  2. (2) $c_{\tau (\hat I_0)^{\vee [(2n-2r)^r]},\tau (\hat J_0)^{\vee [(2n-2r)^r]}}^{\tau (\hat K_0)}=1$ ,

where $\hat I=I\cap [n]\cup \{i-2(m-n)\,:\,i\in \bar I\cap [m+1,2m]\}$ , and $\hat J$ and $\hat K$ are defined similarly?

Proof of Theorem 1.3. Fix an inequality $(A,A^{\prime},B,B^{\prime},C,C^{\prime})$ from (4). It is irredundant for the full-dimensional cone

\begin{equation*}{\operatorname {NL-sat}}(n)={{\mathfrak {sp}}\operatorname {-sat}}(2n)\cap ({\operatorname {Par}}_n)^3\subset {{\mathbb {R}}}^{3n}.\end{equation*}

Thus, it has to appear in Proposition 5.2 for $m=2n$ . Let $(\tilde I,\tilde J,\tilde K)\in \operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (\tilde r,4n))^3$ be the associated Schubert triple. Set $\tilde A^{\prime}=\tilde I\cap [2n]$ , $\tilde A =\overline {\tilde I}\cap [2n]$ , etc. Since $(\tilde I,\tilde J,\tilde K)\in \operatorname {Schub}^{P_0}({\operatorname {Gr}}_\omega (\tilde r,4n))^3$ , $\tilde A^{\prime}, \tilde B^{\prime}, \tilde C^{\prime}\subset [n]$ (by (27)). Thus, comparing (4) and (36), we have

\begin{equation*} \begin{array}{rl@{\quad}rl@{\quad}rl} A&=\tilde A\cap [n],&B&=\tilde B\cap [n],&C&=\tilde C\cap [n],\\[7pt] A^{\prime}&=\tilde A^{\prime}\cap [n]=\tilde A^{\prime},&B^{\prime}&=\tilde B^{\prime}\cap [n]=\tilde B^{\prime},&C^{\prime}&=\tilde C^{\prime}\cap [n]=\tilde C^{\prime}. \end{array} \end{equation*}

Now, Proposition 5.2(1) and Theorem 1.2(2) imply that $r=\tilde r$ . In particular, $|\tilde A|+|\tilde A^{\prime}|=|A|+|A^{\prime}|=r$ and $A=\tilde A$ . Similarly, $B=\tilde B$ and $C=\tilde C$ .

Let $\alpha$ be the simple root of ${\operatorname {Sp}}(4n,{\mathbb {C}})$ associated to $r$ . Observe that the Levi subgroup of $P^\alpha$ has type $A_{r-1}\times C_{2n-r}$ . Let $u,v,w\in W^{P^\alpha }$ corresponding to $(\tilde A^{\prime},\tilde A), (\tilde B^{\prime},\tilde B)$ and $(\tilde C^{\prime},\tilde C)$ , respectively. Proposition 5.2 and its proof show that (34) holds with $P=Q=R=P_0$ . In particular, one can apply the reduction rule proved in [Reference RothRoth11, Theorem 3.1] or [Reference RessayreRes21, Theorem 1]: ${\operatorname {mult}}_{\lambda ,\mu ,\nu }^{2n}$ is a tensor multiplicity for the Levi subgroup of $P^\alpha$ of type $A_{r-1}\times C_{2n-r}$ . The factor $c_{\lambda _{A,A^{\prime}},\mu _{B,B^{\prime}}}^{\nu _{C,C^{\prime}}^*}$ in the theorem corresponds to the factor of type $A_{r-1}$ . Adding zeros, consider $\lambda$ as an element of ${\operatorname {Par}}_{2n}$ . Then the dominant weights to consider for the factor $C_{2n-r}$ are $\lambda _{[2n]-(\tilde A\cup \tilde A^{\prime})}, \mu _{[2n]-(\tilde B\cup \tilde B^{\prime})}, \nu _{[2n]-(\tilde C\cup \tilde C^{\prime})}$ . Since these partitions have length at most $n-r$ , the tensor multiplicity for the factor of $C_{2n-r}$ is a Newell–Littlewood coefficient. The theorem follows.

6. Application to Conjecture 1.4

Corollary 6.1 (Of Theorem 1.2). Conjecture 1.4 holds for $n\leqslant 5$ .

The proof is computational and uses the software Normaliz [Reference Bruns, Ichim, Söger and von der OheBIS].

Fix $n\geqslant 2$ and consider the cone ${{\mathfrak {sp}}\operatorname {-sat}}(n)$ . Consider the two lattices $\Lambda ={{\mathbb {Z}}}^{3n}$ and

\begin{equation*} \Lambda _2=\{(\lambda ,\mu ,\nu )\in ({{\mathbb {Z}}}^n)^3 : |\lambda |+|\mu |+|\nu |\mbox { is even}\}. \end{equation*}

Then ${\operatorname {NL-semigroup}}(n)\subset \Lambda _2\cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$ . Conjecture 1.4 asserts that the converse inclusion holds. The set $\Lambda _2\cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$ is a semigroup of $\Lambda _2$ defined by a family of linear inequalities (explicitly given by Theorem 1.2). Using Normaliz [Reference Bruns, Ichim, Söger and von der OheBIS] one can compute (for small $n$ ) the minimal set of generators, that is, the Hilbert basis, for this semigroup. Hence, to prove Corollary 6.1 one can proceed as follows.

  1. (1) Compute the list of inequalities given by Theorem 1.2.

  2. (2) Compute the Hilbert basis of $\Lambda _2\cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$ using Normaliz.

  3. (3) Check $N_{\lambda ,\mu ,\nu }\gt 0$ for any $(\lambda ,\mu ,\nu )$ in the Hilbert basis.

Table 1 summarizes our computations; see [Reference Gao, Orelowitz, Ressayre and YongGO21].

Table 1. Data for $\Lambda _2\cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$

In the column ‘No. facets’ there are the number of partition inequalities (like $\lambda _1\geqslant \lambda _2$ ) plus the number of inequalities (4) given by Theorem 1.2. The next column counts the inequalities (9) given by applying Theorem 1.5. The number of extremal rays of the cone ${{\mathfrak {sp}}\operatorname {-sat}}(n)$ is also given. The two last column are the cardinalities of the Hilbert bases of the two semigroups $\Lambda _2\cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$ and $\Lambda \cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$ .

7. Littlewood–Richardson coefficients

We recall material [Reference FultonFul97, Reference Fulton and HarrisFH91] on Littlewood–Richardson coefficients and their role in representation theory of the general linear group.

7.1 Representations of ${\operatorname {GL}}(n,{\mathbb {C}})$

The irreducible rational representations $V(\lambda )$ of ${\operatorname {GL}}(n,{\mathbb {C}})$ are indexed by their highest weight

\begin{equation*}\lambda \in \Lambda _n^+=\{(\lambda _1\geqslant \cdots \geqslant \lambda _n)\,:\,\lambda _i\in {{\mathbb {Z}}}\}\supset {\operatorname {Par}}_n.\end{equation*}

One has tensor product multiplicities $c_{\lambda ,\mu }^{\nu }$ defined for any $\lambda ,\mu ,\nu \in \Lambda _n^+$ by

(37) \begin{align} V(\lambda )\otimes V(\mu )=\bigoplus _{\nu \in \Lambda _n^+} V(\nu )^{\oplus c_{\lambda ,\mu }^\nu }. \end{align}

When $\lambda ,\mu ,\nu \in {\operatorname {Par}}_n$ , $c_{\lambda ,\mu }^\nu$ is the Littlewood–Richardson coefficient (which is why we use the same notation).

The dual representation $V(\lambda )^*$ has highest weight

\begin{equation*}\lambda ^*=(-\lambda _n\geqslant \cdots \geqslant -\lambda _1)\in \Lambda _n^+.\end{equation*}

Moreover, for any $a\in {{\mathbb {Z}}}$ ,

(38) \begin{align} V(\lambda +a^n)=({\mathrm {det}})^a\otimes V(\lambda ). \end{align}

Consequently, for any $\lambda ,\mu ,\nu \in \Lambda ^+_n$ ,

(39) \begin{align} c_{\lambda ,\mu }^\nu =c_{\lambda +a^n,\mu +b^n}^{\nu +(a+b)^n}=c_{\lambda ^*+a^n,\mu ^*+b^n}^{\nu ^*+(a+b)^n}; \end{align}

this is [Reference Briand, Orellana and RosasBOR15, Theorem 4]. For $a$ and $b$ big enough, formula (39) implies that $c_{\lambda ,\mu }^\nu$ is a Littlewood–Richardson coefficient.

Let $\nu ^t$ denote the conjugate of $\nu$ . Since $c^{\nu }_{\lambda ,\mu } = c^{\nu ^t}_{\lambda ^t,\mu ^t}$ , by (39),

(40) \begin{align} c^{\nu }_{\lambda ,\mu } = c^{\nu ^t}_{\lambda ^t,\mu ^t} = c^{(\nu ^t)^{\vee [(n+m)^{a+b}]}}_{(\lambda ^t)^{\vee [n^{a+b}]},(\mu ^t)^{\vee [m^{a+b}]}} = c^{(\nu ^{\vee [(a+b)^{n+m}]})^t}_{(\lambda ^{\vee [(a+b)^n]})^t,(\mu ^{\vee [(a+b)^m]})^t} = c^{\nu ^{\vee [(a+b)^{n+m}]}}_{\lambda ^{\vee [(a+b)^n]},\mu ^{\vee [(a+b)^m]}}, \end{align}

for any $m\geqslant \ell (\mu )$ .

7.2 Six-fold Newell–Littlewood coefficients

Let $p$ , $q$ and $m$ be positive integers such that $p+q\leqslant m$ . Following R. Howe, Tan and Willenbring [Reference Howe, Tan and WillenbringHT05], to any $\lambda ^+\in {\operatorname {Par}}_p$ and $\lambda ^-\in {\operatorname {Par}}_q$ we associate the following element in $\Lambda _m^+$ :

\begin{equation*} [\lambda ^+,\lambda ^-]_m = (\lambda ^+_1,\lambda ^+_2,\ldots , \lambda ^+_p,\underbrace {0,\ldots ,0}_{m-p-q},-\lambda ^-_{q},\ldots ,-\lambda ^-_1).\end{equation*}

Let $V(\lambda )\boxtimes V(\mu )$ be the irreducible representation of ${\operatorname {GL}}(n,{\mathbb {C}})\times {\operatorname {GL}}(n,{\mathbb {C}})$ , where $\boxtimes$ refers to the external tensor product. View ${\operatorname {GL}}(n,{\mathbb {C}})\subset {\operatorname {GL}}(n,{\mathbb {C}})\times {\operatorname {GL}}(n,{\mathbb {C}})$ under the diagonal embedding. The associated branching coefficient is

\begin{equation*}[V(\nu ): V(\lambda )\boxtimes V(\mu )]:= \dim {\operatorname {Hom}}_{{\operatorname {GL}}(n,{\mathbb {C}})}(V(\nu ), V(\lambda )\boxtimes V(\mu )|_{{\operatorname {GL}}(n,{\mathbb {C}})}).\end{equation*}

Proposition 7.1 ([Reference Howe, Tan and WillenbringHT05, § 2.1.1], [Reference KingKin71]). Let $\lambda ^\pm$ , $\mu ^\pm$ and $\nu ^\pm$ be six partitions. Let $p,q,r$ and $s$ be four nonnegative integers such that

\begin{equation*} \begin{array}{l@{\quad}l@{\quad}l} \ell (\lambda ^+)\leqslant p,& \ell (\mu ^+)\leqslant r,&\ell (\nu ^+)\leqslant p+r,\\ \ell (\lambda ^-)\leqslant q,&\ell (\mu ^-)\leqslant s,&\ell (\nu ^-)\leqslant q+s.\\ \end{array} \end{equation*}

Let $m$ be a positive integer such that $m\geqslant p+q+r+s$ . Then

\begin{align*} N_{\mu ^+,\nu ^+,\lambda ^+,\mu ^-,\nu ^-,\lambda ^-}= & \ \dim {\operatorname {Hom}}_{{\operatorname {GL}}(n,{\mathbb {C}})}(V(\nu ), V(\lambda )\boxtimes V(\mu ))\\ = & \ c_{[\lambda ^+,\lambda ^-]_m, [\mu ^+,\mu ^-]_m}^{[\nu ^+,\nu ^-]_m}. \end{align*}

Proof. The first equality is in [Reference Howe, Tan and WillenbringHT05, § 2.1.1], which credits [Reference KingKin71]. The second statement follows from [Reference Fulton and HarrisFH91, p. 427].Footnote 3

Conversely, any Littlewood–Richardson coefficient is a six-fold Newell–Littlewood coefficient. More precisely, $c_{\lambda ,\mu }^\nu =N_{\mu ,\nu ,\lambda ,\emptyset ,\emptyset ,\emptyset }$ , which corresponds to the case when $\lambda ^-=\mu ^-=\nu ^-=\emptyset$ in Proposition 7.1.

We now use Proposition 7.1 to rephrase Theorem 1.5. Fix $A$ , $A^{\prime}$ , $B$ , $B^{\prime}$ , $C$ and $C^{\prime}$ subseets of $[n]$ satisfying the two first conditions of Theorem 1.5. Set $p=|B^{\prime}|$ , $r=|C^{\prime}|$ , $q=n-p$ and $s=n-q$ . Observe that $|A|=p+r$ , $|B|\leqslant q$ , $|C|\leqslant s$ and $|A^{\prime}|\leqslant n-p-r\leqslant p+q$ . Finally, set $m=2n=p+q+r+s$ . Since $\ell (\tau (A))\leqslant |A|$ , Proposition 7.1 implies that

\begin{equation*} N_{\tau (A), \tau (C^{\prime}), \tau (B), \tau (A^{\prime}), \tau (C), \tau (B^{\prime})}=c_{[\tau (B^{\prime}),\tau (B)]_m, [\tau (C^{\prime}),\tau (C)]_m}^{[\tau (A),\tau (A^{\prime})]_m} \end{equation*}

is a Littlewood–Richardson coefficient for ${\operatorname {GL}}_{2n}({\mathbb {C}})$ . In particular, in Theorem 1.5, condition $(3)$ can be replaced by:

  1. (3) $c_{[\tau (B^{\prime}),\tau (B)]_m, [\tau (C^{\prime}),\tau (C)]_m}^{[\tau (A),\tau (A^{\prime})]_m}\gt 0.$

We now observe that Proposition 7.1 and Knutson–Tao saturation [Reference Knutson and TaoKT99] imply the saturation result for the six-fold Newell–Littlewood-coefficients from the introduction (Proposition 1.6).

8. Extended Horn inequalities and the proof of Theorem 1.5

8.1 Extended Horn inequalities

We recall the following notion from [Reference Gao, Orelowitz and YongGOY20b].

Definition 8.1. An extended Horn inequality on ${\operatorname {Par}}_n^3$ is

(41) \begin{align} 0\leqslant |\lambda _A|-|\lambda _{A^{\prime}}|+|\mu _B|-|\mu _{B^{\prime}}|+|\nu _C|-|\nu _{C^{\prime}}| \end{align}

where $A,A^{\prime},B,B^{\prime},C,C^{\prime} \subseteq [n]$ satisfy the following assertions:

  1. (I) $A \cap A^{\prime}= B \cap B^{\prime} = C \cap C^{\prime} = \emptyset$ ;

  2. (II) $|A| = |B^{\prime}|+|C^{\prime}|,|B| =|A^{\prime}|+|C^{\prime}|, |C| = |A^{\prime}|+|B^{\prime}|$ ;

  3. (III) there exists $A_1,A_2,B_1,B_2,C_1,C_2\subseteq [n]$ such that

    1. (i) $|A_1|=|A_2|=|A^{\prime}|, |B_1|=|B_2|=|B^{\prime}|, |C_1|=|C_2|=|C^{\prime}|$ ,

    2. (ii) $c^{\tau (A^{\prime})}_{\tau (A_1),\tau (A_2)}, c^{\tau (B^{\prime})}_{\tau (B_1),\tau (B_2)}, c^{\tau (C^{\prime})}_{\tau (C_1),\tau (C_2)}\gt 0$ ,

    3. (iii) $c^{\tau (A)}_{\tau (B_1),\tau (C_2)}, c^{\tau (B)}_{\tau (C_1),\tau (A_2)}, c^{\tau (C)}_{\tau (A_1),\tau (B_2)}\gt 0.$

Definition 8.2. The extended Horn cone is

(42) \begin{align} {\operatorname {EH}}(n):=\{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n^{{\mathbb {Q}}})^3\,:\, \mbox {inequalities (41) are satisfied}\}. \end{align}

Let

\begin{equation*}\overline {{\operatorname {EH}}}(n)={\operatorname {EH}}(n)\cap \{(\lambda ,\mu ,\nu )\in ({\operatorname {Par}}_n)^3: |\lambda |+|\mu |+|\nu |\mbox { is even}\}.\end{equation*}

Conjecture 8.3 [Reference Gao, Orelowitz and YongGOY20b, Conjecture 1.4]. If $(\lambda ,\mu ,\nu )\in {\overline {\operatorname {EH}}}(n)$ then $N_{\lambda ,\mu ,\nu }\gt 0$ .

We will prove a weakened version of Conjecture 8.3.

Theorem 8.4 (Cf. [Reference Gao, Orelowitz and YongGOY20b, Conjecture 1.4]). We have ${\operatorname {EH}}(n)={\operatorname {NL-sat}}(n)$ .

Consequently, we are able to answer an issue raised in [Reference Gao, Orelowitz and YongGOY20b, § 1].

Corollary 8.5. Conjecture 1.4 implies Conjecture 8.3 .

Corollary 8.5 is analogous to the situation in Zelevinsky’s [Reference ZelevinskyZel99], before [Reference Knutson and TaoKT99].

The following shows that Theorem 1.5 is equivalent to Theorem 8.4.

Lemma 8.6. A sextuple $(A,A^{\prime},B,B^{\prime},C,C^{\prime})$ of subsets of $[n]$ parametrizes an extended Horn inequality if and only if it appears in Theorem 1.5 .

Proof. Definition 8.1 implies

\begin{equation*}c_{\tau (B_1),\tau (C_2)}^{\tau (A)} c_{\tau (C_2), \tau (C_1)}^{\tau (C^{\prime})} c_{\tau (C_1), \tau (A_2)}^{\tau (B)} c_{\tau (A_2), \tau (A_1)}^{\tau (A^{\prime})} c_{\tau (A_1), \tau (B_2)}^{\tau (C)} c_{\tau (B_2), \tau (B_1)}^{\tau (B^{\prime})}\gt 0.\end{equation*}

Since $\tau (A_1),\,\tau (A_2)\dots$ have length at most $n$ , this implies $N_{\tau (A),\tau (C^{\prime}),\tau (B),\tau (A^{\prime}),\tau (C),\tau (B^{\prime})}\neq 0$ , and thus $(\tau (A), \tau (C^{\prime}), \tau (B), \tau (A^{\prime}), \tau (C), \tau (B^{\prime}))\in \operatorname {NL}^6\!\operatorname {-sat}(r)$ .

Conversely, if $(\tau (A), \tau (C^{\prime}), \tau (B), \tau (A^{\prime}), \tau (C), \tau (B^{\prime}))\in \operatorname {NL}^6\!\operatorname {-sat}(r)$ , by Proposition 1.6, $N_{\tau (A),\tau (C^{\prime}),\tau (B),\tau (A^{\prime}),\tau (C),\tau (B^{\prime})}\neq 0$ . Therefore, there exists $\alpha _1,\alpha _2,\ldots ,\alpha _6\in {\operatorname {Par}}_n$ such that

\begin{equation*}c_{\alpha _1,\alpha _2}^{\tau (A)}c_{\alpha _2,\alpha _3}^{\tau (C^{\prime})}c_{\alpha _3,\alpha _4}^{\tau (B)} c_{\alpha _4,\alpha _5}^{\tau (A^{\prime})}c_{\alpha _5,\alpha _6}^{\tau (C)}c_{\alpha _6,\alpha _1}^{\tau (B^{\prime})}\gt 0.\end{equation*}

Set $a=|A|$ . Then, the Young diagram of $\tau (A)$ is contained in the rectangle $a\times (n-a)$ . But the nonvanishing of $c_{\alpha _1,\alpha _2}^{\tau (A)}$ implies that $\alpha _1\subset \tau (A)$ . Hence, there exists $B_1\subseteq [n]$ such that $\tau (B_1)=\alpha _1$ . Similarly, we can pick $C_2,C_1,A_2,A_1,B_2\subseteq [n]$ such that $\tau (C_2)=\alpha _2, \tau (C_1)=\alpha _3$ , etc., that satisfy Definition 8.1.

8.2 Proof of Theorem 1.5

( $\Rightarrow$ ) By Lemma 8.6, ${{\operatorname {EH}}}(n)$ is the cone defined by the inequalities in Theorem 1.5. Now, ${\operatorname {NL-sat}}(n)\subseteq {{\operatorname {EH}}}(n)$ is immediate from [Reference Gao, Orelowitz and YongGOY20b, Theorem 1].

( $\Leftarrow$ ) Fix an inequality (4) associated to $(A,A^{\prime},B,B^{\prime},C,C^{\prime})$ appearing in Theorem 1.2. We now show the even stronger statement that

(43) \begin{align} N_{\tau (A),\tau (C^{\prime}),\tau (B),\tau (A^{\prime}),\tau (C),\tau (B^{\prime})}\neq 0. \end{align}

This would imply that the inequality appears in Theorem 1.5, completing the proof.

Set $a=|A|, a^{\prime}=|A^{\prime}|, b=|B|, b^{\prime}=|B^{\prime}|, c=|C|, c^{\prime}=|C^{\prime}|$ and $r=|A|+|A^{\prime}|$ . Let $I\in \operatorname {Schub}({\operatorname {Gr}}_\omega (r,2n))$ be associated to $(A,A^{\prime})\in \operatorname {Schub}^{\prime}({\operatorname {Gr}}_\omega (r,2n))$ , under (20). Similarly define $J$ and $K$ . By (24) and condition (3) in Theorem 1.2, the semigroup property of nonzero Littlewood–Richardson coefficients implies

(44) \begin{align} c^{\tau (K)}_{\tau (I)^{\vee [(2n-r)^r]},\tau (J)^{\vee [(2n-r)^r]}}\neq 0. \end{align}

Fix a nonnegative integer $k$ . Note that $c=a^{\prime}+b^{\prime}$ by condition (2) in Theorem 1.2, and $c_{(a^{\prime})^k,(b^{\prime})^k}^{(c^k)}=1$ . Using the semigroup property once more, one gets

(45) \begin{align} c^{\tau (K)+(c^k)}_{\tau (I)^{\vee [(2n-r)^r]+((a^{\prime})^k)},\tau (J)^{\vee [(2n-r)^r]}+((b^{\prime})^k)}\gt 0. \end{align}

Observing Figure 1,

(46) \begin{align} (\tau (I)^\vee )^t = [\tau (A)^t, \tau (A^{\prime})^t]_{2n-r}+((a^{\prime})^{2n-r}). \end{align}

Similarly,

(47) \begin{align} (\tau (I)^\vee +(a^{\prime})^k)^t = [\tau (A)^t, \tau (A^{\prime})^t]_{m}+((a^{\prime})^{m}) \end{align}

and

(48) \begin{align} (\tau (K)+(c^k))^t = [\tau (C^{\prime})^t, \tau (C)^t]_{m}+(c^{m}), \end{align}

where $m = 2n-r+k$ . Now, by (39), (40), (47) and (48), condition (45) implies

(49) \begin{align} c^{[\tau (C^{\prime})^t, \tau (C)^t]_{m}}_{[\tau (A)^t, \tau (A^{\prime})^t]_m, [\tau (B)^t, \tau (B^{\prime})^t]_m}\gt 0. \end{align}

On the other hand, for $k$ (and hence $m$ ) that is big enough, we can apply Proposition 7.1 to get

(50) \begin{align} N_{\tau (A)^t,\tau (C^{\prime})^t,\tau (B)^t,\tau (A^{\prime})^t,\tau (C)^t,\tau (B^{\prime})^t} = c^{[\tau (C^{\prime})^t,\tau (C)^t]_{m}}_{[\tau (A)^t,\tau (A^{\prime})^t]_{m}, [\tau (B)^t,\tau (B^{\prime})^t]_{m}}. \end{align}

Since the six-fold Newell–Littlewood coefficient are invariant by conjugating the partitions, (49) and (50) imply (43) as expected.

Remark 8.7. The earlier version of this work (arXiv:2107.03152v1) did not use Proposition 7.1. We gave a combinatorial proof, perhaps of independent interest, that connects the celebrated Robinson–Schensted–Knuth algorithm to the ‘demotion’ algorithm of [Reference Gao, Orelowitz and YongGOY20a].

Acknowledgements

We thank Winfried Bruns and the Normaliz team. We also thank the anonymous referee for their helpful remarks.

Conflicts of interest

None.

Financial support

SG, GO, and AY were partially supported by NSF RTG grant DMS 1937241. SG was partially supported by an NSF graduate research fellowship. AY was supported by a Simons collaboration grant and UIUC’s Center for Advanced Study.

Data availability

The computations in § 6 were based on Normaliz [Reference Bruns, Ichim, Söger and von der OheBIS]. The computed data using that software is [Reference Gao, Orelowitz, Ressayre and YongGO21]. Both the software and the data are freely available for download at the indicated locations.

Journal information

Compositio Mathematica is owned by the Foundation Compositio Mathematica and published by the London Mathematical Society in partnership with Cambridge University Press. All surplus income from the publication of Compositio Mathematica is returned to mathematics and higher education through the charitable activities of the Foundation, the London Mathematical Society and Cambridge University Press.

Footnotes

1 We remark that since $(\lambda ,\mu ,\nu )\in {\operatorname {LR-sat}}(n)$ is also in ${\operatorname {NL-sat}}(n)$ if and only if $|\lambda |+|\mu |=|\nu |$ (see [Reference Gao, Orelowitz and YongGOY20a, Lemma 2.2(II)]), LR-sat $(n)$ is a facet of the ${\mathfrak {sp}}_{2n}$ -sat(n).

2 $N_{\lambda _1,\dots ,\lambda _s}$ also has a uniform representation-theoretic interpretation. Discussion may appear elsewhere.

3 Using (38) one can equate this with a Littlewood–Richardson coefficient $c_{\tilde \lambda ,\tilde \mu }^{\tilde \nu }$ where $\tilde \lambda ,\tilde \mu ,\tilde \nu \in {\operatorname {Par}}_m$ .

References

Belkale, P. and Kumar, S., Eigenvalue problem and a new product in cohomology of flag varieties, Invent. Math. 166 (2006), 185228.CrossRefGoogle Scholar
Belkale, P. and Kumar, S., Eigencone, saturation and Horn problems for symplectic and odd orthogonal groups, J. Algebraic Geom. 19 (2010), 199242.CrossRefGoogle Scholar
Briand, E., Orellana, R. and Rosas, M., Rectangular symmetries for coefficients of symmetric functions, Electron. J. Combin. 22 (2015), 3.15.CrossRefGoogle Scholar
Bruns, W., Ichim, B., Söger, C. and von der Ohe, U., Normaliz. Algorithms for rational cones and affine monoids, https://www.normaliz.uni-osnabrueck.de.Google Scholar
Derksen, H. and Weyman, J., The combinatorics of quiver representations, Ann. Inst. Fourier 61 (2011), 10611131.CrossRefGoogle Scholar
Fulton, W., Young tableaux, London Mathematical Society Student Texts, vol. 35 (Cambridge University Press, Cambridge, 1997).Google Scholar
Fulton, W., Eigenvalues, invariant factors, highest weights, and Schubert calculus, Bull. Amer. Math. Soc. (N.S.) 37 (2000), 209249.CrossRefGoogle Scholar
Fulton, W. and Harris, J., Representation theory: A first course. Readings in mathematics, Graduate Texts in Mathematics, vol. 129 (Springer-Verlag, New York, 1991).Google Scholar
Gao, S., Orelowitz, G., Ressayre, N. and Yong, A., Sage programs and data on the NL-semigroup (2021), http://math.univ-lyon1.fr/~ressayre/Info/computedDatumNL.tar.gz.Google Scholar
Gao, S., Orelowitz, G. and Yong, A., Newell-Littlewood numbers, Trans. Amer. Math. Soc. 374 (2021), 63316366.CrossRefGoogle Scholar
Gao, S., Orelowitz, G. and Yong, A., Newell-Littlewood numbers II: Extended Horn inequalities, Algebr. Comb. 5 (2022), 12871297.Google Scholar
Heckman, G. J., Projections of orbits and asymptotic behavior of multiplicities for compact connected Lie groups, Invent. Math. 67 (1982), 333356.CrossRefGoogle Scholar
Howe, R., Tan, E.-C. and Willenbring, J. F., Stable branching rules for classical symmetric pairs, Trans. Amer. Math. Soc. 357 (2005), 16011626.CrossRefGoogle Scholar
King, R. C., Modification rules and products of irreducible representations of the unitary, orthogonal, and symplectic groups, J. Math. Phys. 12 (1971), 15881598.CrossRefGoogle Scholar
King, R. C., Tollu, C. and Toumazet, F., Factorisation of Littlewood-Richardson coefficients, J. Combin. Theory Ser. A 116 (2009), 314333.CrossRefGoogle Scholar
Klyachko, A. A., Stable bundles, representation theory and Hermitian operators, Selecta Math. (N.S.) 4 (1998), 419445.CrossRefGoogle Scholar
Knutson, A. and Tao, T., The honeycomb model of $\textrm {GL}_n({{{\mathbb {C}}}})$ tensor products. I. Proof of the saturation conjecture, J. Amer. Math. Soc. 12 (1999), 10551090.CrossRefGoogle Scholar
Koike, K. and Terada, I., Young-diagrammatic methods for the representation theory of the classical groups of type $B_n,\;C_n,\;D_n$ , J. Algebra 107 (1987), 466511.CrossRefGoogle Scholar
Kumar, S., A survey of the additive eigenvalue problem, Transform. Groups 19 (2014), 10511148.CrossRefGoogle Scholar
Kumar, S., Additive eigenvalue problem, Eur. Math. Soc. Newsl. 98 (2015), 2027.Google Scholar
Littlewood, D. E., Products and plethysms of characters with orthogonal, symplectic and symmetric groups, Canad. J. Math. 10 (1958), 1732.CrossRefGoogle Scholar
Mumford, D., Fogarty, J. and Kirwan, F., Geometric invariant theory, third edition (Springer Verlag, New York, 1994).CrossRefGoogle Scholar
Newell, M. J., Modification rules for the orthogonal and symplectic groups, Proc. Roy. Irish Acad. Sect. A 54 (1951), 153163.Google Scholar
Pasquier, B. and Ressayre, N., The saturation property for branching rules—examples, Exp. Math. 22 (2013), 299312.CrossRefGoogle Scholar
Ressayre, N., Geometric invariant theory and generalized eigenvalue problem, Invent. Math. 180 (2010), 389441.CrossRefGoogle Scholar
Ressayre, N., Multiplicative formulas in Schubert calculus and quiver representation, Indag. Math. (N.S.) 22 (2011), 87102.CrossRefGoogle Scholar
Ressayre, N., A cohomology-free description of eigencones in types A, B, and C, Int. Math. Res. Not. IMRN 21 (2012), 49665005.CrossRefGoogle Scholar
Ressayre, N., Reductions for branching coefficients, J. Lie Theory 31 (2021), 885896.Google Scholar
Richmond, E., A multiplicative formula for structure constants in the cohomology of flag varieties, Michigan Math. J. 61 (2012), 317.CrossRefGoogle Scholar
Roth, M., Reduction rules for Littlewood-Richardson coefficients, Int. Math. Res. Not. IMRN 18 (2011), 41054134.Google Scholar
Zelevinsky, A., Littlewood-Richardson semigroups, in New perspectives in algebraic combinatorics (Berkeley, CA, 1996–97), Mathematical Sciences Research Institute Publications, vol. 38 (Cambridge University Press, Cambridge, 1999), 337345.Google Scholar
Figure 0

Figure 1. $\tau (I)$, $\tau (A)$ and $\tau (A^{\prime})\; (\tau (A^{\prime})\subseteq \tau (I))$.

Figure 1

Table 1. Data for $\Lambda _2\cap {{\mathfrak {sp}}\operatorname {-sat}}(n)$