2.1. Lie type A

Throughout this article, we will employ integer sequences $\alpha =(\alpha _1,\alpha _2,\ldots )$ , which are assumed to have finite support, and we identify with integer vectors. The integer sequence $\alpha $ is a composition if $\alpha _j\geq 0$ for all j. A weakly decreasing composition is called a partition. If $\lambda $ is a partition, the length of $\lambda $ is the integer $\ell (\lambda ):=\#\{i\ |\ \lambda _i\neq 0\}$ , and the conjugate of $\lambda $ is the partition $\lambda '$ with $\lambda ^{\prime }_j:=\#\{i\ |\ \lambda _i\geq j\}$ for all $j\geq 1$ . As is customary, we identify partitions with their Young diagrams of boxes, arranged in left justified rows. An inclusion $\mu \subset \lambda $ of partitions corresponds to the containment of their respective diagrams; in this case, the skew diagram $\lambda /\mu $ is the set-theoretic difference $\lambda \smallsetminus \mu $ . For each integer $r\geq 1$ , let $\delta _r:=(r,r-1,\ldots ,1)$ , $\delta ^{\vee }_r:=(1,2,\ldots ,r)$ , and set $\delta _0:=0$ . Denote by $\epsilon _r$ the sequence whose rth term is 1 and all other terms are zero.

The symmetric group $S_n$ is generated by the simple transpositions $s_i=(i,i+1)$ for $1\leq i \leq n-1$ . There is a natural embedding of $S_n$ in $S_{n+1}$ by adjoining $n+1$ as a fixed point, and we let $S_\infty :=\cup _n S_n$ . We will write a permutation ${\varpi }\in S_n$ using one line notation, as the word $({\varpi }_1,\ldots ,{\varpi }_n)$ where ${\varpi }_i={\varpi }(i)$ .

The length of a permutation ${\varpi }$ , denoted $\ell ({\varpi })$ , is the least integer r such that we have an expression ${\varpi }=s_{i_1} \dots s_{i_r}$ . The word $s_{i_1}\dots s_{i_r}$ is called a reduced decomposition for ${\varpi }$ . An element ${\varpi }\in S_\infty $ has a left descent (respectively, a right descent) at position $i\geq 1$ if $\ell (s_i{\varpi })<\ell ({\varpi })$ (respectively, if $\ell ({\varpi } s_i)<\ell ({\varpi })$ ). The permutation ${\varpi }=({\varpi }_1,{\varpi }_2,\ldots )$ has a right descent at i if and only if ${\varpi }_i>{\varpi }_{i+1}$ , and a left descent at i if and only if ${\varpi }^{-1}(i)>{\varpi }^{-1}(i+1)$ .

The code $\gamma =\gamma ({\varpi })$ of a permutation ${\varpi }\in S_n$ is the sequence $\{\gamma _i\}$ with $\gamma _i:=\#\{j>i\ |\ {\varpi }_j<{\varpi }_i\}$ . The code $\gamma $ determines ${\varpi }$ , as follows. We have ${\varpi }_1=\gamma _1+1$ , and for $i>1$ , ${\varpi }_i$ is the $(\gamma _i+1)$ st element in the complement of $\{{\varpi }_1,\ldots ,{\varpi }_{i-1}\}$ in the sequence $(1,\ldots ,n)$ . Following [Reference Lascoux and Schützenberger18], [Reference Macdonald20], the shape $\lambda =\lambda ({\varpi })$ of ${\varpi }$ is the partition whose parts are the non-zero entries $\gamma _i$ of the code $\gamma ({\varpi })$ , arranged in weakly decreasing order. We have $|\lambda |:=\sum _i \lambda _i=\sum _i \gamma _i = \ell ({\varpi })$ .

For any integer $p\geq 0$ and sequence of variables $Z:=(z_1,z_2,\ldots )$ , the elementary and complete symmetric functions $e_p(Z)$ and $h_p(Z)$ are defined by the generating series

$$\begin{align*}\prod_{i=1}^{\infty}(1+z_it) = \sum_{p=0}^{\infty} e_p(Z)t^p \ \ \ \text{and} \ \ \ \prod_{i=1}^{\infty}(1-z_it)^{-1} = \sum_{p=0}^{\infty} h_p(Z)t^p, \end{align*}$$

respectively. If $r\geq 1,$ then we let $e^r_p(Z):=e_p(z_1,\ldots ,z_r)$ and $h^r_p(Z):=h_p(z_1,\ldots ,z_r)$ denote the polynomials obtained from $e_p(Z)$ and $h_p(Z)$ by setting $z_i=0$ for all $i>r$ . Let $e^0_p(Z)=h^0_p(Z):=\delta _{0p}$ , where $\delta _{0p}$ denotes the Kronecker delta, and for $r<0$ , define $h^r_p(Z):=e^{-r}_p(Z)$ and $e^r_p(Z):=h^{-r}_p(Z)$ .

Let $X:=(x_1,x_2,\ldots )$ and $Y:=(y_1,y_2,\ldots )$ be two sequences of independent variables. There is an action of $S_\infty $ on ${\mathbb Z}[X,Y]$ by ring automorphisms, defined by letting the simple reflections $s_i$ act by interchanging $x_i$ and $x_{i+1}$ while leaving all the remaining variables fixed. Define the divided difference operator $\partial _i^x$ on ${\mathbb Z}[X,Y]$ by

$$\begin{align*}\partial_i^xf := \frac{f-s_if}{x_i-x_{i+1}} \ \ \ \text{for } i\geq 1. \end{align*}$$

Consider the ring involution $\pi :{\mathbb Z}[X,Y]\to {\mathbb Z}[X,Y]$ determined by $\pi (x_i)=-y_i$ and $\pi (y_i)=-x_i$ for each i, and set $\partial _i^y:=\pi \partial _i^x\pi $ .

For any $p,r,s\in {\mathbb Z}$ , define the polynomial ${}^rh^s_p$ by

$$\begin{align*}{}^rh^s_p:=\sum_{i=0}^p h^r_i(X)e_{p-i}^s(-Y). \end{align*}$$

We have the following basic lemma.

Lemma 1. Suppose that $p,r,s\in {\mathbb Z}$ . For all $i\geq 1$ , we have

$$\begin{align*}\partial^x_i ({}^rh_p^s)= \begin{cases} {}^{r+1}h_{p-1}^s & \text{if } r=\pm i, \\ 0 & \text{otherwise} \end{cases} \ \quad \mathrm{and} \ \quad \partial^y_i ({}^rh_p^s)= \begin{cases} {}^rh_{p-1}^{s-1} & \text{if } s=\pm i, \\ 0 & \text{otherwise}. \end{cases} \end{align*}$$

The double Schubert polynomials ${\mathfrak S}_{\varpi }$ for ${\varpi }\in S_{\infty }$ of Lascoux and Schützenberger [Reference Lascoux17], [Reference Lascoux and Schützenberger18] are the unique family of polynomials in ${\mathbb Z}[X,Y]$ such that

(4) $$ \begin{align} \partial_i^x{\mathfrak S}_{\varpi} = \begin{cases} {\mathfrak S}_{{\varpi} s_i} & \text{if } \ell({\varpi} s_i)<\ell({\varpi}), \\ 0 & \text{otherwise}, \end{cases} \quad \partial_i^y{\mathfrak S}_{\varpi} = \begin{cases} {\mathfrak S}_{s_i{\varpi}} & \text{if } \ell(s_i{\varpi})<\ell({\varpi}), \\ 0 & \text{otherwise}, \end{cases} \end{align} $$

for all $i\geq 1$ , together with the condition that the constant term of ${\mathfrak S}_{\varpi }$ is $1$ if ${\varpi }=1$ , and $0$ otherwise.

2.2. Lie type C

The Weyl group for the root system of type $\text {C}_n$ is the group of signed permutations on the set $\{1,\ldots ,n\}$ , denoted $W_n$ . The group $W_n$ is generated by the simple transpositions $s_i=(i,i+1)$ for $1\leq i \leq n-1$ together with the sign change $s_0$ , which fixes all $j\in [2,n]$ and sends $1$ to $\overline {1}$ (a bar over an integer here means a negative sign). We write the elements of $W_n$ as n-tuples $(w_1,\ldots , w_n)$ , where $w_i:=w(i)$ for each $i\in [1,n]$ . There is a natural embedding of $W_n$ in $W_{n+1}$ by adjoining $n+1$ as a fixed point, and we let $W_\infty :=\cup _n W_n$ . The symmetric groups $S_n$ and $S_\infty $ are the subgroups of $W_n$ and $W_\infty $ , respectively, generated by the reflections $s_i$ for i positive. The length $\ell (w)$ and the reduced decompositions of an element $w\in W_\infty $ is defined as in type A. We have

$$\begin{align*}\ell(w) = \#\{i<j\ |\ w_i>w_j\} + \sum_{i\, :\, w_i <0}|w_i| \end{align*}$$

for every $w\in W_\infty $ .

An element $w\in W_\infty $ has a right descent (respectively, a left descent) at position $i\geq 0$ if $\ell (ws_i)<\ell (w)$ (respectively, if $\ell (s_iw)<\ell (w)$ ). The signed permutation $w=(w_1,w_2,\ldots )$ has a right descent at 0 if and only if $w_1<0$ , and a right descent at $i \geq 1$ if and only if $w_i>w_{i+1}$ . The element w has a left descent at 0 if and only if $w^{-1}(1)<0$ , that is, $w=(\cdots \overline {1} \cdots )$ . The element w has a left descent at $i \geq 1$ if and only if $w^{-1}(i)>w^{-1}(i+1)$ , that is, w has one of the following four forms:

$$\begin{align*}(\cdots i+1 \cdots i \cdots), \quad (\cdots i \cdots \overline{i+1} \cdots), \quad (\cdots \overline{i+1} \cdots i \cdots), \quad (\cdots \overline{i} \cdots \overline{i+1} \cdots). \end{align*}$$

Let $w\in W_\infty $ be a signed permutation. Following [Reference Tamvakis30, Definition 2], the strict partition $\mu =\mu (w)$ is the one whose parts are the absolute values of the negative entries of w, arranged in decreasing order. The A-code of w is the sequence $\gamma =\gamma (w)$ with $\gamma _i:=\#\{j>i\ |\ w_j<w_i\}$ . We define a partition $\nu =\nu (w)$ by

$$\begin{align*}\nu_j = \#\{i\ |\ \gamma_i\geq j\}, \ \ \text{for all } j\geq 1. \end{align*}$$

Finally, the shape of w is the partition $\lambda (w):=\mu (w)+\nu (w)$ . The element w is uniquely determined by $\mu (w)$ and $\gamma (w)$ , and we have $|\lambda (w)|=\ell (w)$ .

Example 1. (a) For the signed permutation $w := (\overline {5},3, \overline {4}, 7, \overline {1}, \overline {6}, 2)$ in $W_7$ , we obtain $\mu = (6,5,4,1)$ , $\gamma =(1, 4, 1, 3, 1,0,0)$ , $\nu = (5,2, 2, 1)$ , and $\lambda = (11, 7, 6, 2)$ .

(b) Let $k\geq 0$ . An element $w\in W_\infty $ is k-Grassmannian if $\ell (ws_i)>\ell (w)$ for all $i\neq k$ . This is equivalent to the conditions

$$\begin{align*}0<w_1<\cdots < w_k \quad \mathrm{and} \quad w_{k+1}<w_{k+2}<\cdots. \end{align*}$$

If w is a k-Grassmannian element of $W_\infty $ , then $\lambda (w)$ is the k-strict partition associated with w in [Reference Buch, Kresch and Tamvakis7, Section 6.1].

(c) Suppose that the first right descent of $w\in W_n$ is $k\geq 0$ , and let $m=\ell (\mu )$ and $\ell =\ell (\lambda )$ . Then, $\mu $ is a strict partition and $\nu \subset k^{n-k}+\delta _{n-k-1}$ , with $\nu _j\geq k$ for all $j\in [1,m]$ . It follows that

$$\begin{align*}\lambda_1>\lambda_2>\cdots > \lambda_m >\max(\lambda_{m+1},k) \geq \lambda_{m+1}\geq \lambda_{m+2}\geq \cdots \geq \lambda_{\ell}. \end{align*}$$

Lemma 2 [Reference Macdonald20], [Reference Tamvakis30]

If $i\geq 1$ , $w\in W_\infty $ , and $\gamma =\gamma (w)$ , then

$$\begin{align*}\gamma_i>\gamma_{i+1} \Leftrightarrow w_i>w_{i+1} \Leftrightarrow \ell(ws_i)=\ell(w) -1. \end{align*}$$

If any of the above conditions hold, then

$$\begin{align*}\gamma(ws_i) = (\gamma_1,\ldots, \gamma_{i-1}, \gamma_{i+1},\gamma_i-1,\gamma_{i+2},\gamma_{i+3},\ldots). \end{align*}$$

Let $c:=(c_1,c_2,\ldots )$ be a sequence of commuting variables, and set $c_0:=1$ and $c_p:=0$ for $p<0$ . Consider the graded ring $\Gamma $ which is the quotient of the polynomial ring ${\mathbb Z}[c]$ modulo the ideal generated by the relations

(5) $$ \begin{align} c_pc_p+2\sum_{i=1}^p(-1)^ic_{p+i}c_{p-i}=0, \ \ \ \text{for all } p\geq 1. \end{align} $$

Let $X:=(x_1,x_2,\ldots )$ and $Y:=(y_1,y_2,\ldots )$ be two sequences of variables. Following [Reference Billey and Haiman3], [Reference Ikeda, Mihalcea and Naruse13], there is an action of $W_\infty $ on $\Gamma [X,Y]$ by ring automorphisms, defined as follows. The simple reflections $s_i$ for $i\geq 1$ act by interchanging $x_i$ and $x_{i+1}$ while leaving all the remaining variables fixed. The reflection $s_0$ maps $x_1$ to $-x_1$ , fixes the $x_j$ for $j\geq 2$ and all the $y_j$ , and satisfies

$$\begin{align*}s_0(c_p) := c_p+2\sum_{j=1}^p x_1^jc_{p-j} \ \ \text{for all } p\geq 1. \end{align*}$$

For each $i\geq 0$ , define the divided difference operator $\partial _i^x$ on $\Gamma [X,Y]$ by

$$\begin{align*}\partial_0^xf := \frac{f-s_0f}{-2x_1}, \qquad \partial_i^xf := \frac{f-s_if}{x_i-x_{i+1}} \ \ \ \text{for } i\geq 1. \end{align*}$$

Consider the ring involution $\varphi :\Gamma [X,Y]\to \Gamma [X,Y]$ determined by

$$\begin{align*}\varphi(x_j) = -y_j, \qquad \varphi(y_j) = -x_j, \qquad \varphi(c_p)=c_p \end{align*}$$

and set $\partial _i^y:=\varphi \partial _i^x\varphi $ for each $i\geq 0$ . The right and left divided difference operators $\partial ^x_i$ and $\partial ^y_i$ on $\Gamma [X,Y]$ satisfy the right and left Leibnitz rules

(6) $$ \begin{align} \partial^x_i(fg) = (\partial^x_if)g+(s_if)\partial^x_ig \quad \ \mathrm{and} \quad \ \partial^y_i(fg) = (\partial^y_if)g+(s^y_if)\partial^y_ig, \end{align} $$

where $s_i^y:=\varphi s_i \varphi $ , for every $i\geq 0$ .

For any $p,r,s\in {\mathbb Z}$ , define the polynomial ${}^rc^s_p$ by

$$\begin{align*}{}^rc^s_p:=\sum_{i=0}^p\sum_{j=0}^p c_{p-i-j}e^{r}_i(X)h_j^s(-Y). \end{align*}$$

We have the following basic lemma, which stems from [Reference Ikeda and Matsumura12, Section 5.1].

Lemma 3. (a) Suppose that $p,r,s\in {\mathbb Z}$ . For all $i\geq 0$ , we have

$$\begin{align*}\partial^x_i ({}^rc_p^s)= \begin{cases} {}^{r-1}c_{p-1}^s & \text{if } r=\pm i, \\ 0 & \text{otherwise}. \end{cases} \ \quad \mathrm{and} \ \quad \partial^y_i ({}^rc_p^s)= \begin{cases} {}^rc_{p-1}^{s+1} & \text{if } s=\pm i, \\ 0 & \text{otherwise}. \end{cases} \end{align*}$$

(b) For all $i\geq 1$ , $r,s\geq 0$ , and indices p and q, we have

$$\begin{align*}\partial^y_i({}^rc_p^{-i}\,{}^sc_q^i) = {}^rc_{p-1}^{-i+1}\,{}^sc_q^{i+1} + {}^rc_p^{-i+1}\,{}^sc_{q-1}^{i+1}. \end{align*}$$

Suppose $r,s\geq 0$ , and let ${\mathfrak c}_p:={}^rc^{-s}_p$ for each $p\in {\mathbb Z}$ . We then have the relations

(7) $$ \begin{align} {\mathfrak c}_p{\mathfrak c}_p+2\sum_{i=1}^p (-1)^i {\mathfrak c}_{p+i}{\mathfrak c}_{p-i}=0 \ \ \text{for all} \ \, p> r+s \end{align} $$

in $\Gamma [X,Y]$ . Indeed, if ${\mathcal C}(t):=\sum _{p=0}^{\infty } {\mathfrak c}_pt^p$ is the generating function for the ${\mathfrak c}_p$ , we have

$$\begin{align*}{\mathcal C}(t) = \prod_{i=1}^r(1+x_it)\prod_{j=1}^s(1-y_jt) \left(\sum_{p=0}^{\infty}c_pt^p\right) \end{align*}$$

and hence

$$\begin{align*}{\mathcal C}(t){\mathcal C}(-t) = \prod_{i=1}^r(1-x^2_it^2)\prod_{j=1}^s(1-y^2_jt^2), \end{align*}$$

which is a polynomial in t of degree $2(r+s)$ .

The type C double Schubert polynomials ${\mathfrak C}_w$ for $w\in W_{\infty }$ of Ikeda, Mihalcea, and Naruse [Reference Ikeda, Mihalcea and Naruse13] are the unique family of elements of $\Gamma [X,Y]$ such that

(8) $$ \begin{align} \partial_i^x{\mathfrak C}_w = \begin{cases} {\mathfrak C}_{ws_i} & \text{if } \ell(ws_i)<\ell(w), \\ 0 & \text{otherwise}, \end{cases} \quad \partial_i^y{\mathfrak C}_w = \begin{cases} {\mathfrak C}_{s_iw} & \text{if } \ell(s_iw)<\ell(w), \\ 0 & \text{otherwise}, \end{cases} \end{align} $$

for all $i\geq 0$ , together with the condition that the constant term of ${\mathfrak C}_w$ is $1$ if $w=1$ , and $0$ otherwise.

2.3. Lie types B and D

When working with the orthogonal Lie types, we use coefficients in the ring ${\mathbb Z}[\frac {1}{2}]$ . For any $w\in W_\infty $ , the type B double Schubert polynomial ${\mathfrak B}_w$ of [Reference Ikeda, Mihalcea and Naruse13] satisfies ${\mathfrak B}_w=2^{-s(w)}{\mathfrak C}_w$ , where $s(w)$ is the number of indices j such that $w_j<0$ . The odd orthogonal case is therefore entirely similar to the symplectic case. In the rest of this section, we provide the corresponding preliminaries for the even orthogonal group, that is, in Lie type D, and assume that $n\geq 2$ .

The Weyl group $\widetilde {W}_n$ for the root system $\text {D}_n$ is the subgroup of $W_n$ consisting of all signed permutations with an even number of sign changes. The group $\widetilde {W}_n$ is an extension of $S_n$ by the element $s_\Box =s_0s_1s_0$ , which acts on the right by

$$\begin{align*}(w_1,w_2,\ldots,w_n)s_\Box=(\overline{w}_2,\overline{w}_1,w_3,\ldots,w_n). \end{align*}$$

There is a natural embedding $\widetilde {W}_n\hookrightarrow \widetilde {W}_{n+1}$ of Weyl groups, induced by the embedding $W_n\hookrightarrow W_{n+1}$ , and we let $\widetilde {W}_\infty := \cup _n \widetilde {W}_n$ . The elements of the set ${\mathbb N}_\Box :=\{\Box ,1,\ldots \}$ index the simple reflections in $\widetilde {W}_\infty $ . The length $\ell (w)$ and reduced decompositions of an element $w\in \widetilde {W}_\infty $ are defined as before. We have

$$\begin{align*}\ell(w) = \#\{i<j\ |\ w_i>w_j\} + \sum_{i\, :\, w_i <0}(|w_i|-1) \end{align*}$$

for every $w\in \widetilde {W}_\infty $ .

An element $w\in \widetilde {W}_\infty $ has a right descent (respectively, a left descent) at position $i\in {\mathbb N}_\Box $ if $\ell (ws_i)<\ell (w)$ (respectively, if $\ell (s_iw)<\ell (w)$ ). The element $w=(w_1,w_2,\ldots )$ has a right descent at $\Box $ if and only if $w_1<-w_2$ , and a right descent at $i\geq 1$ if and only if $w_i>w_{i+1}$ . We use the notation $\widehat {1}$ to denote $1$ or $\overline {1}$ , determined by the parity of the number of negative entries of w. The following result corrects [Reference Tamvakis28, Lemma 4].

Lemma 4. Suppose that w is an element of $\widetilde {W}_\infty $ .

(a) We have $\ell (s_\Box w)<\ell (w)$ if and only if w has one of the following four forms:

$$\begin{align*}(\cdots \widehat{1} \cdots \overline{2} \cdots), \quad (\cdots \overline{2} \cdots \overline{1} \cdots), \quad (\cdots 2 \cdots \overline{1} \cdots). \end{align*}$$

(b) Assume that $i\geq 1$ . We have $\ell (s_iw)<\ell (w)$ if and only if w has one of the following four forms:

$$\begin{align*}(\cdots i+1 \cdots i \cdots), \quad (\cdots i \cdots \overline{i+1} \cdots), \quad (\cdots \overline{i+1} \cdots i \cdots), \quad (\cdots\overline{i} \cdots \overline{i+1} \cdots). \end{align*}$$

There is an involution $\iota :\widetilde {W}_\infty \to \widetilde {W}_\infty $ which interchanges $s_\Box $ and $s_1$ ; we have $\iota (w) = s_0ws_0$ in the hyperoctahedral group $W_\infty $ . We deduce that $\iota (w)=w$ if and only if w has type 0, while if w has positive type and $|w_r|=1$ for some $r>1$ , then

$$\begin{align*}\iota(w)=(-w_1,w_2,\ldots,w_{r-1},-w_r,w_{r+1},\ldots). \end{align*}$$

It follows that $\iota $ interchanges type 1 and type 2 elements. The next definition refines the notion of the shape of an element of $\widetilde {W}_\infty $ introduced in [Reference Tamvakis30, Definition 5].

Observe that the element w is uniquely determined by $\mu (w)$ , $\gamma (w)$ , and ${\mathrm {type}}(w)$ . Moreover, we have $|\lambda (w)|=\ell (w)$ .

Example 2. (a) For the signed permutation $w := (3, 2, \overline {7}, 1, 5, 4, \overline {6})$ in $\widetilde {W}_7$ , we obtain $\mu = (6,5)$ , $\gamma =(4,3, 0, 1, 2, 1, 0)$ , $\nu = (5, 3, 2,1)$ , and $\lambda = (11, 8, 2,1)$ with ${\mathrm {type}}(\lambda )=1$ . The element $\iota (w)= (\overline {3}, 2, \overline {7}, \overline {1}, 5, 4, \overline {6})$ has shape $\overline {\lambda } = (11, 8, 2,1)$ with ${\mathrm {type}}(\overline {\lambda })=2$ . Both w and $\iota (w)$ have primary index $k=1$ .

(b) Let $k\geq 1$ . An element w of $\widetilde {W}_\infty $ is k-Grassmannian if $\ell (ws_i)>\ell (w)$ for all $i\neq k$ , if $k>1$ , and for all $i\notin \{\Box ,1\}$ , if $k=1$ . This is equivalent to the conditions

$$\begin{align*}|w_1|<w_2<\cdots < w_k \quad \mathrm{and} \quad w_{k+1}<w_{k+2}<\cdots, \end{align*}$$

the first condition being vacuous if $k=1$ . If w is a k-Grassmannian element of $\widetilde {W}_\infty $ , then $\lambda (w)$ is the typed k-strict partition associated with w in [Reference Buch, Kresch and Tamvakis6, Section 6.1].

(c) Suppose that the primary index of $w\in \widetilde {W}_n$ is $k\geq 1$ , and let $m=\ell (\mu )$ and $\ell =\ell (\lambda )$ . Then, $\mu $ is a strict partition and $\nu \subset k^{n-k}+\delta _{n-k-1}$ , with $\nu _j \geq k$ for all $j\in [1,m]$ . We therefore have

$$\begin{align*}\lambda_1>\lambda_2>\cdots > \lambda_m >\max(\lambda_{m+1},k) \geq \lambda_{m+1}\geq \lambda_{m+2}\geq \cdots \geq \lambda_{\ell}. \end{align*}$$

Let $b:=(b_1,b_2,\ldots )$ be a sequence of commuting variables, and set $b_0:=1$ and $b_p:=0$ for $p<0$ . Consider the graded ring $\Gamma '$ which is the quotient of the polynomial ring ${\mathbb Z}[b]$ modulo the ideal generated by the relations

$$\begin{align*}b_pb_p+2\sum_{i=1}^{p-1}(-1)^i b_{p+i}b_{p-i}+(-1)^p b_{2p}=0, \ \ \ \text{for all } p\geq 1. \end{align*}$$

We regard $\Gamma $ as a subring of $\Gamma '$ via the injective ring homomorphism which sends $c_p$ to $2b_p$ for every $p\geq 1$ .

Following [Reference Billey and Haiman3], [Reference Ikeda, Mihalcea and Naruse13], we define an action of $\widetilde {W}_\infty $ on $\Gamma '[X,Y]$ by ring automorphisms as follows. The simple reflections $s_i$ for $i \geq 1$ act by interchanging $x_i$ and $x_{i+1}$ and leaving all the remaining variables fixed. The reflection $s_\Box $ maps $(x_1,x_2)$ to $(-x_2,-x_1)$ , fixes the $x_j$ for $j\geq 3$ and all the $y_j$ , and satisfies, for any $p\geq 1$ ,

$$ \begin{align*} s_\Box(b_p) := b_p+(x_1+x_2)\sum_{j=0}^{p-1}\left(\sum_{a+b=j}x_1^ax_2^b\right) c_{p-1-j}. \end{align*} $$

For each $i\in {\mathbb N}_\Box $ , define the divided difference operator $\partial _i^x$ on $\Gamma '[X,Y]$ by

$$\begin{align*}\partial_\Box^xf := \frac{f-s_\Box f}{-x_1-x_2}, \qquad \partial_i^xf := \frac{f-s_if}{x_i-x_{i+1}} \ \ \ \text{for } i\geq 1. \end{align*}$$

Consider the ring involution $\varphi ':\Gamma '[X,Y]\to \Gamma '[X,Y]$ determined by

$$\begin{align*}\varphi'(x_j) = -y_j, \qquad \varphi'(y_j) = -x_j, \qquad \varphi'(b_p)=b_p \end{align*}$$

and set $\partial _i^y:=\varphi '\partial _i^x\varphi '$ for each $i\in {\mathbb N}_\Box $ . The right and left divided difference operators $\partial ^x_i$ and $\partial ^y_i$ on $\Gamma '[X,Y]$ satisfy the right and left Leibnitz rules

(9) $$ \begin{align} \partial^x_i(fg) = (\partial^x_if)g+(s_if)\partial^x_ig \quad \ \mathrm{and} \quad \ \partial^y_i(fg) = (\partial^y_if)g+(s^y_if)\partial^y_ig, \end{align} $$

where $s_i^y:=\varphi ' s_i \varphi '$ , for every $i\in {\mathbb N}_\Box $ .

Let $r\geq 0$ and set $\displaystyle {}^rc_p:= \sum _{i=0}^p c_{p-i}h^{-r}_i(X)$ . Define ${}^rb_p := {}^rc_p$ for $p<r$ , $\displaystyle {}^rb_p := \frac {1}{2}{}^rc_p$ for $p>r$ , and set

$$\begin{align*}{}^rb_r := \frac{1}{2}{}^rc_r + \frac{1}{2}e^r_r(X) \quad \text{and} \quad {}^r\widetilde{b}_r := \frac{1}{2}{}^rc_r - \frac{1}{2}e^r_r(X). \end{align*}$$

For $s\in \{0,1\}$ , let $\displaystyle {}^ra^s_p:=\frac {1}{2}{}^rc_p + \sum _{i=1}^p {}^rc_{p-i}h^s_i(-Y)$ , and define

$$\begin{align*}{}^rb^s_r:={}^rb_r + \sum_{i=1}^r {}^rc_{r-i}h^s_i(-Y), \quad \text{and} \quad {}^r\widetilde{b}^s_r:={}^r\widetilde{b}_r + \sum_{i=1}^r {}^rc_{r-i}h^s_i(-Y). \end{align*}$$

We have the following propositions, which are proved as in [Reference Tamvakis28, Section 2].

Proposition 1. Suppose that $p,q\in {\mathbb Z}$ and $r,s\geq 1$ .

(a) For all $i \geq 1$ , we have

$$\begin{align*}\partial^x_i ({}^rc_p^q)= \begin{cases} {}^{r-1}c_{p-1}^q & \text{if } r=\pm i, \\ 0 & \text{otherwise} \end{cases} \quad \ and \ \quad \partial^y_i ({}^rc_p^q)= \begin{cases} {}^rc_{p-1}^{q+1} & \text{if } q=\pm i, \\ 0 & \text{otherwise}. \end{cases} \end{align*}$$

We have

$$\begin{align*}\partial^y_\Box \left({}^rc_p^q\right)= \begin{cases} {}^rc_{p-1}^2 & \text{if } q=1, \\ 2\left({}^rc_{p-1}^2\right) & \text{if } q=0, \\ 2\left({}^rc_{p-1}^1\right) -{}^rc_{p-1} & \text{if } q=-1, \\ 0 & \text{if } |q|\geq 2. \end{cases} \end{align*}$$

(b) For all $i\geq 1$ , we have

$$\begin{align*}\partial^y_i({}^rc_p^{-i}\,{}^sc_q^i) = {}^rc_{p-1}^{-i+1}\,{}^sc_q^{i+1} + {}^rc_p^{-i+1}\,{}^sc_{q-1}^{i+1}. \end{align*}$$

We also require certain variations of the above identities. Let $f_r$ be an indeterminate of degree r, which will equal ${}^rb_r$ , ${}^r\widetilde {b}_r$ , or $\displaystyle \frac {1}{2}\,{}^rc_r$ in the sequel. We also let $f_0 \in \{0,1\}$ . For any $p,s\in {\mathbb Z}$ , define ${}^r\widehat {c}_p^{\,s}$ by

$$\begin{align*}{}^r\widehat{c}_p^{\,s}:= {}^rc_p^s + \begin{cases} (2f_r-{}^rc_r)e^{p-r}_{p-r}(-Y) & \text{if } s = r - p < 0, \\ 0 & \text{otherwise}. \end{cases} \end{align*}$$

For $s\in \{0,1\}$ , define

$$\begin{align*}f_r^s:= f_r+\sum_{j=1}^r {}^rc_{r-j}h_j^s(-Y), \end{align*}$$

set $\widetilde {f}_r:={}^rc_r-f_r$ and $\widetilde {f}_r^s:={}^rc_r-2f_r+f_r^s$ .

Proposition 2. Suppose that $p\in {\mathbb Z}$ and $p>r$ .

(a) For all $i \geq 1$ , we have

$$\begin{align*}\partial^x_i({}^r\widehat{c}_p^{\,r-p}) = \begin{cases} {}^{r-1}\widehat{c}_{p-1}^{\,r-p} & \text{if } i=p-r\geq 2, \\ 2\varphi'(f_r) & \text{if } i=p-r=1, \\ 0 & \text{otherwise} \end{cases} \end{align*}$$

and

$$\begin{align*}\partial^y_i ({}^r\widehat{c}_p^{\,r-p})= \begin{cases} {}^r\widehat{c}_{p-1}^{\,r-p+1} & \text{if } i=p-r\geq 2, \\ 2f_r & \text{if } i=p-r= 1, \\ 0 & \text{otherwise}. \end{cases} \end{align*}$$

We have

$$\begin{align*}\partial^y_\Box \left({}^r\widehat{c}_p^{\,r-p}\right)= \begin{cases} 2\widetilde{f}^1_r & \text{if } r-p=-1, \\ 0 & \text{if } r-p < -1. \end{cases} \end{align*}$$

(b) For all $i\geq 2$ , we have

$$\begin{align*}\partial^y_i({}^r\widehat{c}_p^{\,-i}\,{}^sc_q^i) = {}^r\widehat{c}_{p-1}^{\,-i+1}\,{}^sc_q^{i+1} + {}^r\widehat{c}_p^{\,-i+1}\,{}^sc_{q-1}^{i+1}. \end{align*}$$

Fix $r,s\geq 0$ , and define ${\mathfrak c}_p:={}^rc^{-s}_p$ for each $p\in {\mathbb Z}$ . For $p=r+s$ , set ${\mathfrak d}_p:=e^r_r(X)e^s_s(-Y)$ . Then, in addition to the relations (7), we have the relations

(10) $$ \begin{align} ({\mathfrak c}_p+{\mathfrak d}_p)({\mathfrak c}_p-{\mathfrak d}_p)+ 2\sum_{i=1}^p (-1)^i {\mathfrak c}_{p+i}{\mathfrak c}_{p-i}=0 \ \ \text{for} \ \, p = r+s \end{align} $$

in $\Gamma '[X,Y]$ .

Following [Reference Ikeda, Mihalcea and Naruse13], the type D double Schubert polynomials ${\mathfrak D}_w$ for $w\in \widetilde {W}_{\infty }$ are the unique family of elements of $\Gamma '[X,Y]$ satisfying the equations

(11) $$ \begin{align} \partial_i^x{\mathfrak D}_w = \begin{cases} {\mathfrak D}_{ws_i} & \text{if } \ell(ws_i)<\ell(w), \\ 0 & \text{otherwise}, \end{cases} \quad \partial_i^y{\mathfrak D}_w = \begin{cases} {\mathfrak D}_{s_iw} & \text{if } \ell(s_iw)<\ell(w), \\ 0 & \text{otherwise}, \end{cases} \end{align} $$

for all $i\in {\mathbb N}_\Box $ , together with the condition that the constant term of ${\mathfrak D}_w$ is $1$ if $w=1$ , and $0$ otherwise.

3.1. Alternating properties in types A–C

For each $r\geq 1$ , let ${\sigma }^r=({\sigma }^r_i)_{i\in {\mathbb Z}}$ be a sequence of variables, with ${\sigma }^r_0=1$ and ${\sigma }^r_i=0$ for each $i<0$ , and let ${\mathbb Z}[{\sigma }]$ denote the polynomial ring in the variables ${\sigma }^r_i$ for $i,r\geq 1$ . For any integer sequence $\alpha $ , let ${\sigma }_\alpha :={\sigma }^1_{\alpha _1}{\sigma }^2_{\alpha _2}\dots $ , and for any raising operator R, set $R\, {\sigma }_\alpha :={\sigma }_{R\alpha }$ .

Fix $j\geq 1$ , let z be a variable, set $\tau ^r:={\sigma }^r$ for each $r\neq j$ and $\tau ^j_p = {\sigma }^j_p+z\,{\sigma }^j_{p-1}$ for each $p\in {\mathbb Z}$ . If $\alpha :=(\alpha _1,\ldots ,\alpha _\ell )$ and $\alpha ':=(\alpha ^{\prime }_1,\ldots ,\alpha ^{\prime }_{\ell '})$ are two integer vectors and $r,s\in {\mathbb Z}$ , we let $(\alpha ,r,s,\alpha ')$ denote the integer vector $(\alpha _1,\ldots ,\alpha _\ell ,r,s,\alpha ^{\prime }_1,\ldots ,\alpha ^{\prime }_{\ell '})$ . The following two lemmas are generalizations of [Reference Buch, Kresch and Tamvakis7, Lemmas 1.2 and 1.3].

Lemma 5. Let $\lambda =(\lambda _1,\ldots ,\lambda _{j-1})$ and $\mu =(\mu _{j+2},\ldots ,\mu _\ell )$ be integer vectors, and D be a valid set of pairs. Assume that ${\sigma }^j={\sigma }^{j+1}$ , $(j,j+1)\notin D$ , and that for each $h<j$ , $(h,j)\in D$ if and only if $(h,j+1)\in D$ .

(a) For any integers r and s, we have

$$\begin{align*}R^D\,{\sigma}_{\lambda,r,s,\mu} = - R^D\, {\sigma}_{\lambda,s-1,r+1,\mu} \end{align*}$$

in ${\mathbb Z}[{\sigma }]$ .

(b) For any integer r, we have

$$\begin{align*}R^D\,\tau_{\lambda,r,r,\mu} = R^D\, {\sigma}_{\lambda,r,r,\mu} \end{align*}$$

in ${\mathbb Z}[{\sigma },z]$ .

Proof. The proof of (a) is identical to that of [Reference Buch, Kresch and Tamvakis7, Lemma 1.2]. For part (b), we use linearity in the jth position to obtain $R\,\tau _{\lambda ,r,r,\mu } =R\, {\sigma }_{\lambda ,r,r,\mu } + z\,R\, {\sigma }_{\lambda ,r-1,r,\mu }$ , for any raising operator R that appears in the expansion of the power series $R^D$ . Adding these equations gives

$$\begin{align*}R^D\,\tau_{\lambda,r,r,\mu} = R^D\, {\sigma}_{\lambda,r,r,\mu} + z\,R^D\, {\sigma}_{\lambda,r-1,r,\mu}. \end{align*}$$

Now, part (a) implies that $R^D\, {\sigma }_{\lambda ,r-1,r,\mu }=0$ .

Fix $k\geq 0$ , let ${\mathfrak c}=({\mathfrak c}_i)_{i\in {\mathbb Z}}$ be another sequence of variables, and consider the relations

(12) $$ \begin{align} {\mathfrak c}_p{\mathfrak c}_p+2\sum_{i=1}^p (-1)^i {\mathfrak c}_{p+i}{\mathfrak c}_{p-i}=0 \ \ \text{for all} \ \, p> k. \end{align} $$

Lemma 6. Let $\lambda =(\lambda _1,\ldots ,\lambda _{j-1})$ and $\mu =(\mu _{j+2},\ldots ,\mu _\ell )$ be integer vectors, and D be a valid set of pairs. Assume that ${\sigma }^j={\sigma }^{j+1}={\mathfrak c}$ , $(j,j+1)\in D$ , and that for each $h>j+1$ , $(j,h)\in D$ if and only if $(j+1,h)\in D$ .

(a) If $r,s\in {\mathbb Z}$ are such that $r+s> 2k$ , then we have

$$\begin{align*}R^D\,{\sigma}_{\lambda,r,s,\mu} = - R^D\,{\sigma}_{\lambda,s,r,\mu} \end{align*}$$

in the ring ${\mathbb Z}[{\sigma }]$ modulo the relations coming from (12).

(b) For any integer $r>k$ , we have

$$\begin{align*}R^D\,\tau_{\lambda,r+1,r,\mu} = R^D\, {\sigma}_{\lambda,r+1,r,\mu} \end{align*}$$

in the ring ${\mathbb Z}[{\sigma },z]$ modulo the relations coming from (12).

Proof. The proof of (a) is identical to that of [Reference Buch, Kresch and Tamvakis7, Lemma 1.3]. For part (b), we expand $R^D\,\tau _{\lambda ,r+1,r,\mu }$ and use linearity in the jth position to obtain

$$\begin{align*}R^D\,\tau_{\lambda,r+1,r,\mu} = R^D\, {\sigma}_{\lambda,r+1,r,\mu} + z\,R^D\, {\sigma}_{\lambda,r,r,\mu}. \end{align*}$$

Now, part (a) implies that $R^D\, {\sigma }_{\lambda ,r,r,\mu }$ vanishes modulo the relations (12).

3.2. Alternating properties in type D

In type D, we will require certain variations of Lemmas 5 and 6. For each $r\geq 1$ , we introduce a new sequence of variables ${\upsilon }^r=({\upsilon }^r_i)_{i\in {\mathbb Z}}$ such that ${\upsilon }^r_i=0$ for each $i\leq 0$ . Let ${\mathbb Z}[{\sigma }, {\upsilon }]$ denote the polynomial ring in the variables ${\sigma }^r_i$ , ${\upsilon }^r_i$ for $i,r\geq 1$ . For each $r\geq 1$ , define the sequence $\widehat {{\sigma }}^r$ by $\widehat {{\sigma }}^r_i:={\sigma }^r_i+(-1)^r{\upsilon }^r_i$ for each i, and for any integer sequence $\alpha $ , let $\widehat {{\sigma }}_\alpha :=\widehat {{\sigma }}^1_{\alpha _1}\widehat {{\sigma }}^2_{\alpha _2}\dots $ .

Fix an integer $d\geq 0$ such that ${\upsilon }^r_i=0$ for all i whenever $r>d$ . If $R:=\prod _{i<j} R_{ij}^{n_{ij}}$ is a raising operator, denote by ${\mathrm {supp}}_d(R)$ the set of all indices i and j such that $n_{ij}>0$ and $j \leq d$ . Let D be a valid set of pairs and R be any raising operator appearing in the expansion of the power series $R^D$ . Let $\lambda =(\lambda _1,\ldots ,\lambda _\ell )$ be any integer vector and set $\rho :=R\lambda $ . Define

$$\begin{align*}R \star \widehat{{\sigma}}_{\lambda} = \overline{{\sigma}}_{\rho} := \overline{{\sigma}}^1_{\rho_1}\dots \overline{{\sigma}}^{\ell}_{\rho_\ell} \end{align*}$$

where, for each $i\geq 1$ and $p\in {\mathbb Z}$ ,

$$\begin{align*}\overline{{\sigma}}^i_p:= \begin{cases} {\sigma}^i_p & \text{if } i\in{\mathrm{supp}}_d(R), \\ \widehat{{\sigma}}^i_p & \text{otherwise}. \end{cases} \end{align*}$$

Fix $j\geq 1$ , set $\widehat {\tau }^i:=\widehat {{\sigma }}^i$ for each $i\neq j$ and $\widehat {\tau }^j_p = \widehat {{\sigma }}^j_p+z\,\widehat {{\sigma }}^j_{p-1}$ for each $p\in {\mathbb Z}$ .

Lemma 7. Let $\lambda =(\lambda _1,\ldots ,\lambda _{j-1})$ and $\mu =(\mu _{j+2},\ldots ,\mu _\ell )$ be integer vectors, and D be a valid set of pairs. Assume that $j>d$ , ${\sigma }^j={\sigma }^{j+1}$ , $(j,j+1)\notin D$ , and that for each $h<j$ , $(h,j)\in D$ if and only if $(h,j+1)\in D$ .

(a) For any integers r and s, we have

$$\begin{align*}R^D\star\widehat{{\sigma}}_{\lambda,r,s,\mu} = - R^D\star \widehat{{\sigma}}_{\lambda,s-1,r+1,\mu} \end{align*}$$

in ${\mathbb Z}[{\sigma },{\upsilon }]$ .

(b) For any integer r, we have

$$\begin{align*}R^D\star\widehat{\tau}_{\lambda,r,r,\mu} = R^D\star \widehat{{\sigma}}_{\lambda,r,r,\mu} \end{align*}$$

in ${\mathbb Z}[{\sigma },{\upsilon },z]$ .

Fix $k\geq 0$ , let ${\mathfrak c}=({\mathfrak c}_i)_{i\in {\mathbb Z}}$ and ${\mathfrak d}=({\mathfrak d}_i)_{i\in {\mathbb Z}}$ be two other sequences of variables such that ${\mathfrak d}_p=0$ for all $p>k+1$ , and consider the relations

(13) $$ \begin{align} ({\mathfrak c}_p+{\mathfrak d}_p)({\mathfrak c}_p-{\mathfrak d}_p)+ 2\sum_{i=1}^p (-1)^i {\mathfrak c}_{p+i}{\mathfrak c}_{p-i}=0 \ \ \text{for all} \ \, p> k. \end{align} $$

Lemma 8. Let $\lambda =(\lambda _1,\ldots ,\lambda _{j-1})$ and $\mu =(\mu _{j+2},\ldots ,\mu _\ell )$ be integer vectors, and D be a valid set of pairs. Assume that $j<d$ , ${\sigma }^j={\sigma }^{j+1}={\mathfrak c}$ , ${\upsilon }^j={\upsilon }^{j+1}={\mathfrak d}$ , $(j,j+1)\in D$ , and that for each $h>j+1$ , $(j,h)\in D$ if and only if $(j+1,h)\in D$ .

(a) If $r,s\in {\mathbb Z}$ are such that $r+s> 2k+2$ , then we have

(14) $$ \begin{align} R^D\star\widehat{{\sigma}}_{\lambda,r,s,\mu} = - R^D\star\widehat{{\sigma}}_{\lambda,s,r,\mu} \end{align} $$

and

(15) $$ \begin{align} R^D\star\widehat{{\sigma}}_{\lambda,k+1,k+1,\mu} = 0 \end{align} $$

in the ring ${\mathbb Z}[{\sigma },{\upsilon }]$ modulo the relations coming from (13).

(b) For any integer $r>k$ , we have

$$\begin{align*}R^D\star\widehat{\tau}_{\lambda,r+1,r,\mu} = R^D\star \widehat{{\sigma}}_{\lambda,r+1,r,\mu} \end{align*}$$

in the ring ${\mathbb Z}[{\sigma },{\upsilon },z]$ modulo the relations coming from (13).

Proof. The proof of (14) is identical to that of [Reference Buch, Kresch and Tamvakis7, Lemma 1.3]. The proof of (15) follows the same argument, using (14) and induction to reduce to the case when $\mu $ is empty. For any integer vector $\rho $ with at most d components, define $T_\rho :=R^D \star \widehat {{\sigma }}_{\rho }$ . If g is the least integer such that $2g\geq \ell $ and $\rho := (\lambda ,r,s)$ , then we have the relation

$$\begin{align*}T_\rho = \sum_{j=2}^{2g} (-1)^j T_{\rho_1,\rho_j} T_{\rho_2,\ldots,\widehat{\rho}_j,\ldots,\rho_{2g}}. \end{align*}$$

The proof is now completed by induction, as in [Reference Buch, Kresch and Tamvakis7, Lemma 1.3]. For part (b), we expand $R^D\star \widehat {\tau }_{\lambda ,r+1,r,\mu }$ and use linearity in the jth position to obtain

$$\begin{align*}R^D\star\widehat{\tau}_{\lambda,r+1,r,\mu} = R^D\star \widehat{{\sigma}}_{\lambda,r+1,r,\mu} + z\,R^D\, \widehat{{\sigma}}_{\lambda,r,r,\mu}. \end{align*}$$

Now, part (a) implies that $R^D\star \widehat {{\sigma }}_{\lambda ,r,r,\mu }$ vanishes modulo the relations (13).

4.1. Definitions and main theorem

As Lie theorists know well, type A is very special when compared to the other Lie types. In the theory of amenable elements, this manifests itself in the fact that we can work with dominant elements instead of leading elements. The result is the simplified treatment given here, which does not have a direct analog in types B–D. Another difference in type A is that the order of application of the divided difference operators is switched: we first use the left divided differences, then the right ones. But by far the main distinction between type A and the other classical types is that one can use Jacobi–Trudi determinants, represented here by $R^{\emptyset }$ , instead of the more general raising operator expressions $R^D$ that define theta and eta polynomials, which are essential ingredients of the theory for the symplectic and orthogonal groups.

If ${\varpi }\in S_n$ and $v\in S_m$ , then ${\varpi }$ is called v-avoiding if ${\varpi }$ does not contain a subword $({\varpi }_{i_1},\ldots , {\varpi }_{i_m})$ having the same relative order as $(v_1,\ldots ,v_m)$ . The notion of v-avoidance also makes sense when ${\varpi }$ is any integer vector $({\varpi }_1,\ldots ,{\varpi }_n)$ with distinct components ${\varpi }_i$ . We say that ${\varpi }$ is dominant if its code $\gamma ({\varpi })$ is a partition, or equivalently, if ${\varpi }$ is $132$ -avoiding (see [Reference Macdonald20, Proposition 1.30] and [Reference Reifegerste22, Theorem 2.2]).

For the next result, we refer to [Reference Knuth15, Exercise 2.2.1.5] and [Reference Stump24, Section 2.2].

For any three integer vectors $\alpha ,\beta ,\rho \in {\mathbb Z}^\ell $ , which we view as integer sequences with finite support, define ${}^{\rho }h^\beta _\alpha :={}^{\rho _1}h^{\beta _1}_{\alpha _1}\,{}^{\rho _2}h^{\beta _2}_{\alpha _2}\dots $ . Given any raising operator $R=\prod _{i<j}R_{ij}^{n_{ij}}$ , let $R\, {}^{\rho }h^\beta _{\alpha } := {}^{\rho }h^\beta _{R\alpha }$ .

Proposition 3 [Reference Macdonald20, Proposition 6.14]

Suppose that ${\varpi }\in S_n$ is dominant. Then, we have

$$\begin{align*}{\mathfrak S}_{\varpi} = R^{\emptyset} \, {}^{\delta^\vee_{n-1}}h^{\lambda({\varpi})}_{\lambda({\varpi})}. \end{align*}$$

Proof. We use descending induction on $\ell ({\varpi })$ . Let ${\varpi }_0:=(n,\ldots ,1)$ denote the longest element in $S_n$ . One knows from [Reference Lascoux17] and [Reference Macdonald20, Proposition 3.5] that the equation

$$\begin{align*}{\mathfrak S}_{{\varpi}_0} = R^{\emptyset}\, {}^{\delta^\vee_{n-1}}h_{\delta_{n-1}}^{\delta_{n-1}} \end{align*}$$

holds in ${\mathbb Z}[X,Y]$ , so the result is true when ${\varpi }={\varpi }_0$ .

Suppose that ${\varpi }\neq {\varpi }_0$ and ${\varpi }$ is dominant of shape $\lambda $ . Then, $\lambda \subset \delta _{n-1}$ and $\lambda \neq \delta _{n-1}$ . Let $r\geq 1$ be the largest integer such that $\lambda _i=n-i$ for $i\in [1,r]$ , and let $j:=\lambda _{r+1}+1={\varpi }_{r+1}\leq n-r-1$ . Then, $s_j{\varpi }$ is dominant of length $\ell ({\varpi })+1$ and $\lambda (s_j{\varpi })=\lambda ({\varpi })+\epsilon _{r+1}$ . Using Lemma 1 and the left Leibnitz rule, we deduce that for any integer sequence $\alpha =(\alpha _1,\ldots ,\alpha _n)$ , we have

$$\begin{align*}\partial^y_j\left({}^{\delta^\vee_{n-1}}h^{\lambda(s_j{\varpi})}_{\alpha}\right) = {}^{\delta^\vee_{n-1}}h^{\lambda({\varpi})}_{\alpha-\epsilon_{r+1}}. \end{align*}$$

We conclude that

$$\begin{align*}{\mathfrak S}_{\varpi} = \partial^y_j({\mathfrak S}_{s_j{\varpi}}) = \partial^y_j\left(R^{\emptyset} \, {}^{\delta^\vee_{n-1}}h^{\lambda(s_j{\varpi})}_{\lambda(s_j{\varpi})}\right)= R^{\emptyset} \, {}^{\delta^\vee_{n-1}}h^{\lambda({\varpi})}_{\lambda({\varpi})}. \end{align*}$$

Definition 5. Let ${\varpi }$ be an amenable permutation with code $\gamma $ and shape $\lambda $ , with $\ell =\ell (\lambda )$ . Define two sequences ${\mathfrak f}={\mathfrak f}({\varpi })$ and ${\mathfrak g}={\mathfrak g}({\varpi })$ of length $\ell $ as follows. For $1\leq j \leq \ell $ , set

$$\begin{align*}{\mathfrak f}_j:=\max(i\ |\ \gamma_i\geq \lambda_j) \end{align*}$$

and let

$$\begin{align*}{\mathfrak g}_j:={\mathfrak f}_{\mathfrak q}+\lambda_{\mathfrak q}-{\mathfrak q}, \end{align*}$$

where ${\mathfrak q}$ is the least integer such that ${\mathfrak q}\geq j$ and $\lambda _{\mathfrak q}>\lambda _{{\mathfrak q}+1}$ . We call ${\mathfrak f}$ the right flag of ${\varpi }$ , and ${\mathfrak g}$ the left flag of ${\varpi }$ .

It is clear from Lemma 2 that the right flag ${\mathfrak f}$ of an amenable permutation is a weakly increasing sequence consisting of right descents of ${\varpi }$ . We will show that the left flag ${\mathfrak g}$ is a weakly decreasing sequence consisting of left descents of ${\varpi }$ .

Proposition 4. Suppose that $\widehat {{\varpi }}\in S_n$ is dominant with $\widehat {\lambda }:=\lambda (\widehat {{\varpi }})$ . Let ${\omega }$ be a $231$ -avoiding permutation such that $\ell (\widehat {{\varpi }}{\omega })=\ell (\widehat {{\varpi }})-\ell ({\omega })$ , and set ${\varpi }:=\widehat {{\varpi }} {\omega }$ , $\gamma :=\gamma ({\varpi })$ , and $\lambda :=\lambda ({\varpi })$ . Then, the sequence $\delta ^\vee _{n-1}+\widehat {\lambda }-\lambda $ is weakly increasing, and

$$\begin{align*}{\mathfrak S}_{\varpi} = R^{\emptyset} \, {}^{\delta^\vee_{n-1}+\widehat{\lambda}-\lambda}h_{\lambda}^{\widehat{\lambda}}. \end{align*}$$

Moreover, if $\lambda _{\mathfrak q}>\lambda _{{\mathfrak q}+1}$ , then $\widehat {\lambda }_{\mathfrak q}$ is a left descent of ${\varpi }$ , ${\mathfrak q}+\widehat {\lambda }_{\mathfrak q}-\lambda _{\mathfrak q}$ is a right descent of ${\varpi }$ , and we have ${\mathfrak q}+\widehat {\lambda }_{\mathfrak q}-\lambda _{\mathfrak q}= \max (i\ |\ \gamma _i\geq \lambda _{\mathfrak q})$ .

Proof. Suppose that $\widehat {{\varpi }}$ is of shape $\widehat {\lambda } = \widehat {\gamma } = (p_1^{n_1},p_2^{n_2}\ldots ,p_t^{n_t})$ , where $p_1>\cdots > p_t$ . Then, the right descents of $\widehat {{\varpi }}$ are at positions $d_1:=n_1, d_2:=n_1+n_2, \ldots , d_t:= n_1+\cdots +n_t$ . Since we have $\widehat {{\varpi }}_j<\widehat {{\varpi }}_{j+1}$ for all $j\neq d_r$ for $r\in [1,t]$ , we deduce that

$$\begin{align*}\widehat{{\varpi}}_1 = p_1+1, \widehat{{\varpi}}_{d_1+1}=p_2+1,\ldots, \widehat{{\varpi}}_{d_{t-1}+1}=p_t+1, \widehat{{\varpi}}_{d_t+1}=1. \end{align*}$$

Moreover, since $\widehat {{\varpi }}$ is $132$ -avoiding, it follows that the left descents of $\widehat {{\varpi }}$ are $p_1,\ldots ,p_t$ . Finally, Proposition 3 gives

(16) $$ \begin{align} {\mathfrak S}_{\widehat{{\varpi}}} = R^{\emptyset} \, {}^{\delta^\vee_{n-1}}h^{\widehat{\lambda}}_{\widehat{\lambda}} \end{align} $$

so the result holds when ${\omega }=1$ and ${\varpi }=\widehat {{\varpi }}$ is dominant.

Suppose next that ${\varpi }:=\widehat {{\varpi }} {\omega }$ for some $231$ -avoiding permutation ${\omega }$ such that $\ell (\widehat {{\varpi }}{\omega })=\ell (\widehat {{\varpi }})-\ell ({\omega })$ . Lemma 9 implies that ${\omega }$ has a reduced decomposition of the form $R_1\dots R_{n-1}$ , where each $R_j$ is a (possibly empty) subword of $s_{n-1}\dots s_1$ and all simple reflections in $R_{p+1}$ are also contained in $R_p$ , for every $p\geq 1$ . Now repeated application of (4), Lemma 1, and the right Leibnitz rule (6) in equation (16) give

$$\begin{align*}{\mathfrak S}_{\varpi}= \partial^x_{{\omega}^{-1}}({\mathfrak S}_{\widehat{{\varpi}}}) = R^{\emptyset} \, {}^{\delta_{n-1}^{\vee} +\widehat{\lambda}-\lambda}h^{\widehat{\lambda}}_{\lambda}. \end{align*}$$

We will show that the sequence $\delta ^\vee _{n-1}+\widehat {\lambda }-\lambda $ is weakly increasing and verifies the last assertion, about the left and right descents of ${\varpi }$ .

Using Lemma 2, we study the right action of the successive simple transpositions in the reduced decomposition $R_1\dots R_{n-1}$ for ${\omega }$ on the code $\widehat {\gamma }$ of $\widehat {{\varpi }}$ . The action of these on $\widehat {\gamma }$ is by a finite sequence of moves $\alpha \mapsto \alpha '$ , where $\alpha :=\gamma (v)$ and $\alpha ':=\gamma (v')$ for some $v,v'\in S_n$ . Here, $v'=vs_{j-1}\dots s_i$ for some $i<j$ such that $\ell (v')=\ell (v)-j+i$ , and $s_{j-1}\dots s_i$ is a subword of some $R_p$ with $j-i$ maximal. Since the initial code $\widehat {\gamma }$ is weakly decreasing, we have $\alpha _i\geq \cdots \geq \alpha _{j-1}>\alpha _j$ , and

$$\begin{align*}\alpha'=(\alpha_1,\ldots,\alpha_{i-1},\alpha_j,\alpha_i-1,\ldots,\alpha_{j-1}-1,\alpha_{j+1}, \alpha_{j+2},\ldots). \end{align*}$$

We say that the move is performed on the interval $[i,j]$ , or is an $[i,j]$ -move. The procedure is defined as the performance of finitely many moves to $\widehat {\gamma }$ , ending in the code $\gamma $ . This describes the effect of multiplying $\widehat {{\varpi }}$ on the right by ${\omega }$ .

Example 3. Suppose that $\widehat {{\varpi }}:=(5,6,7,4,3,8,2,1)$ in $S_8$ with code

$$ \begin{align*}\widehat{\gamma}=(4,4,4,3,2,2,1,0).\end{align*} $$

If ${\omega }:=s_7s_6s_5s_4s_3s_2s_7s_6s_5s_4s_6s_5$ , then $\widehat {{\varpi }} {\omega } = (5,1,6,2,3,7,4,8)$ and $\gamma =\gamma ({\varpi } {\omega }) = (4,0,3,0,0,1,0,0)$ . The procedure from $\widehat {\gamma }$ to $\gamma $ consists of a $[2,8]$ -move, followed by a $[4,8]$ -move, followed by a $[5,7]$ -move:

$$\begin{align*}(4,4,4,3,2,2,1,0) \mapsto (4,0,3,3,2,1,1,0) \mapsto (4,0,3,0,2,1,0,0)\mapsto (4,0,3,0,0,1,0,0). \end{align*}$$

Notice that after an $[i,j]$ -move $\alpha \mapsto \alpha '$ , we have

(17) $$ \begin{align} \alpha^{\prime}_i=\gamma_i=\min(\alpha^{\prime}_r\ |\ r\in [i,j]) \geq \max(\alpha^{\prime}_r\ | \ r>j). \end{align} $$

Let $\mu $ and $\mu '$ be the shapes of v and $v'$ , respectively, and set $f:=\delta ^\vee _{n-1}+\widehat {\lambda }-\mu $ (respectively, $f':=\delta ^\vee _{n-1}+\widehat {\lambda }-\mu '$ ). We then have

$$\begin{align*}\mu'=(\mu_1,\ldots,\mu_{r-1},\mu_r-1,\ldots,\mu_{s-1}-1,\mu_s,\mu_{s+1},\ldots) \end{align*}$$

for some $r<s$ with $s-r=j-i$ , and

$$\begin{align*}f'=(f_1,\ldots,f_{r-1},f_r+1,\ldots,f_{s-1}+1,f_s,f_{s+1},\ldots). \end{align*}$$

Since $\mu _s = \alpha _j \leq \alpha _{j-1}-1 = \mu _{s-1}-1$ , we deduce that $f^{\prime }_s-f^{\prime }_{s-1} = f_s-f_{s-1}-1 = \mu _s-\mu _{s-1}-1\geq 0$ . It follows by induction on the number of moves that the sequence f is weakly increasing.

Suppose that $\mu ^{\prime }_d>\mu ^{\prime }_{d+1}$ for some d. Using (17) and induction on the number of moves, we deduce that $f^{\prime }_d=\max (i\ |\ \alpha ^{\prime }_i\geq \mu ^{\prime }_d)$ . This implies that for any ${\mathfrak q}$ such that $\lambda _{\mathfrak q}>\lambda _{{\mathfrak q}+1}$ , we have ${\mathfrak q}+\widehat {\lambda }_{\mathfrak q}-\lambda _{\mathfrak q} = \max (i\ |\ \gamma _i\geq \lambda _{\mathfrak q})$ , and hence that ${\mathfrak q}+\widehat {\lambda }_{\mathfrak q}-\lambda _{\mathfrak q}$ is a right descent of ${\varpi }$ , in view of Lemma 2.

We claim that $\widehat {\lambda }_d$ is a left descent of $v'$ . Clearly, the left descents of v and $v'$ are subsets of $\{p_1,\ldots , p_t\}$ . There is at most one left descent $p_e$ of v that is not a left descent of $v'$ , and this occurs if and only if $v_j=p_e$ and $v_h=p_e+1$ for some $h\in [i,j-1]$ . Since $\alpha _h\geq \cdots \geq \alpha _{j-1}>\alpha _j$ , we deduce that $\alpha _h=\cdots = \alpha _{j-1}=\alpha _j+1$ , and hence $\mu ^{\prime }_{s-(j-h)+1} = \cdots = \mu ^{\prime }_s=\mu _s$ . We conclude that $\widehat {\lambda }_d\neq p_e$ , completing the proof of the claim, and the proposition.

Example 4. Let $\widehat {{\varpi }}:=(4,5,6,2,1,3)$ , a dominant permutation in $S_6$ with shape $\widehat {\lambda } = \gamma (\widehat {{\varpi }})=(3,3,3,1)$ . Take ${\omega }:=s_4s_3s_2s_1s_4s_3$ in Proposition 4, so that ${\varpi }=\widehat {{\varpi }} {\omega } = (1,4,2,5,6,3)$ , with $\gamma (\widehat {{\varpi }}{\omega })=(0,2,0,1,1,0)$ and $\lambda =(2,1,1)$ . We have $\delta _5^\vee +\widehat {\lambda }-\lambda = (2,4,5,5,5)$ , and deduce that

$$\begin{align*}{\mathfrak S}_{\varpi} = R^{\emptyset} \, {}^{(2,4,5,5,5)}h_{(2,1,1,0)}^{(3,3,3,1)} = R^{\emptyset} \, {}^{(2,4,5)}h_{(2,1,1)}^{(3,3,3)}. \end{align*}$$

Theorem 1. For any amenable permutation ${\varpi }$ , we have

$$\begin{align*}{\mathfrak S}_{\varpi} = R^{\emptyset} \, {}^{{\mathfrak f}({\varpi})}h_{\lambda({\varpi})}^{{\mathfrak g}({\varpi})}. \end{align*}$$

Example 5. Consider the amenable permutation ${\varpi }:=(3,4,6,1,5,2)$ in $S_6$ . We then have $\gamma ({\varpi })=(2,2,3,0,1,0)$ , $\lambda ({\varpi })=(3,2,2,1)$ , ${\mathfrak f}({\varpi })=(3,3,3,5)$ , and ${\mathfrak g}({\varpi })=(5,2,2,2)$ . Theorem 1 gives

$$\begin{align*}{\mathfrak S}_{346152} = R^{\emptyset} \, {}^{(3,3,3,5)}h_{(3,2,2,1)}^{(5,2,2,2)}. \end{align*}$$

Recall from [Reference Lascoux and Schützenberger18], [Reference Lascoux and Schützenberger19] that a permutation ${\varpi }$ is vexillary if and only if it is $2143$ -avoiding. Equivalently, ${\varpi }$ is vexillary if and only if $\lambda ({\varpi }^{-1})=\lambda ({\varpi })'$ .

Proof. According to [Reference Macdonald20, Proposition 1.32], a permutation is vexillary if and only if its code $\gamma $ satisfies the following two conditions, for any $i<j$ : (i) if $\gamma _i\leq \gamma _j$ , then $\gamma _i\leq \gamma _k$ for any k with $i<k<j$ and (ii) if $\gamma _i>\gamma _j$ , then the number of k with $i<k<j$ and $\gamma _k<\gamma _j$ is at most $\gamma _i-\gamma _j$ .

Assume first that ${\varpi }$ is amenable, so that ${\varpi }=\widehat {{\varpi }}{\omega }$ for some dominant permutation $\widehat {{\varpi }}$ and $231$ -avoiding permutation ${\omega }$ . Using Lemma 2, we see that the code $\widehat {\gamma }$ of $\widehat {{\varpi }}$ is transformed into the code $\gamma $ of ${\varpi }$ by the moves of the procedure described in the proof of Proposition 4.

We claim that the sequence $\gamma $ is a vexillary code. It follows from the inequalities (17) that for any $[i,j]$ -move of the procedure, we have $\gamma _s\leq \gamma _i\leq \gamma _r$ for every $r\in [i,j]$ and $s>j$ . Moreover, if $r\neq i$ for all $[i,j]$ -moves of the procedure, then $\gamma _s\leq \gamma _r$ for all $s>r$ . It is easy to see from this that $\gamma $ satisfies the vexillary conditions (i) and (ii). Indeed, choose $r<s$ such that $\gamma _r\leq \gamma _s$ , and some $t\in [r,s]$ . If $r=i$ for some $[i,j]$ -move, then we have $\gamma _r\leq \gamma _k$ for all $k\in [i,j]$ , while if $k>j$ , then we must have $\gamma _r=\gamma _s=\gamma _k$ . Therefore, $\gamma _r\leq \gamma _t$ . If $r\neq i$ for all $[i,j]$ -moves, then $\gamma _r=\gamma _s$ and hence $\gamma _r=\gamma _t$ . To prove (ii), suppose that $r<k<s$ and $\gamma _r>\gamma _s>\gamma _k$ . Then, we must have $k=i$ for some $[i,j]$ -move of the procedure, where $s\leq j$ . We conclude that the number of such k is at most $\gamma _r-\gamma _s$ .

Conversely, suppose that ${\varpi }\in S_n$ is a vexillary permutation with code $\gamma $ . We call an integer $i\geq 1$ an initial index if there exists an $s>i$ with $\gamma _i<\gamma _s$ . We claim that there is a canonical $312$ -avoiding permutation ${\omega }$ such that $\ell ({\varpi }{\omega })= \ell ({\varpi })+\ell ({\omega })$ , ${\varpi }{\omega }$ is dominant, and ${\omega }_a=a$ if $a<i$ for every initial index i. This will complete the proof of the theorem, by applying Lemma 9.

To establish the claim, we argue by descending induction on the length of ${\varpi }$ . Observe that ${\varpi }$ has no initial index if and only if ${\varpi }$ is a dominant permutation. Hence, if ${\varpi }$ is already dominant, then we must take ${\omega }$ to be the identity.

Assume that ${\varpi }$ is not dominant. We say that the index j is associated with the initial index i of ${\varpi }$ if j is the maximum s such that $\gamma _i<\gamma _s$ . Let i be the smallest initial index, let j be associated with i, and set ${\varpi }':={\varpi } s_i\dots s_{j-1}$ . The vexillary condition (i) and Lemma 2 imply that

$$\begin{align*}\gamma({\varpi}') = (\gamma_i,\ldots, \gamma_{i-1}, \gamma_{i+1}+1,\ldots, \gamma_j+1,\gamma_i,\gamma_{j+1},\ldots) \end{align*}$$

and $\ell ({\varpi }')=\ell ({\varpi })+j-i>\ell ({\varpi })$ . It follows by checking conditions (i) and (ii) that ${\varpi }'$ is vexillary and that every initial index $i'$ of ${\varpi }'$ satisfies $i'\geq i$ .

By the inductive hypothesis, there exists a canonical $312$ -avoiding permutation ${\omega }'$ with $\ell ({\varpi }'{\omega }')=\ell ({\varpi }')+\ell ({\omega }')$ , ${\varpi }'{\omega }'$ dominant, and ${\omega }^{\prime }_a=a$ if $a<i'$ for every initial index $i'$ of ${\varpi }'$ . The claim is proved with ${\omega }:=s_i\dots s_{j-1}{\omega }'$ , once we check that ${\omega }$ is $312$ -avoiding. Indeed, since ${\omega }^{\prime }_a=a$ for all $a<i$ and $\ell (s_i\dots s_{j-1}{\omega }')=j-i+\ell ({\omega }')$ , we must have

$$\begin{align*}{\omega}'=(1,2,\ldots,i-1,\ldots,a_1,\ldots,a_2, \ldots, a_{j-i},\ldots,j,\ldots) \end{align*}$$

and

$$\begin{align*}{\omega} = (1,2,\ldots,i-1,\ldots,a_1+1,\ldots,a_2+1, \ldots, a_{j-i}+1,\ldots,i,\ldots) \end{align*}$$

where the set $\{a_1,\ldots ,a_{j-i}\}$ is equal to $\{i,\ldots , j-1\}$ . As ${\omega }'$ is $312$ -avoiding, there are no integers $a<b<c$ such that ${\omega }^{\prime }_a>{\omega }^{\prime }_c >{\omega }^{\prime }_b$ . It is easy to see from this and the above relation between ${\omega }'$ and ${\omega }$ that the latter permutation has the same property, and therefore is also $312$ -avoiding.

Let ${\varpi }$ be a vexillary permutation with code $\gamma $ and shape $\lambda $ , and let ${\omega }$ be the canonical $312$ -avoiding permutation associated with ${\varpi }$ in the proof of Theorem 2. Define a new sequence $\widehat {\gamma }$ by the prescription

$$\begin{align*}\widehat{\gamma}_\alpha:=\gamma_\alpha+\#\{i\ |\ i \text{ is an initial index with associated index } j \text{ and } i<\alpha\leq j\} \end{align*}$$

for each $\alpha \geq 1$ . Let $\widehat {\lambda }$ be the partition obtained by listing the entries of $\widehat {\gamma }$ in weakly decreasing order. Then, $\widehat {\lambda }$ is the shape of ${\varpi } {\omega }$ .

Consider the skew Young diagram $\tau ({\varpi }):=\widehat {\lambda }/\lambda $ . For each $i\geq 1$ , fill the boxes in row i of $\tau ({\varpi })$ with a strictly decreasing sequence of consecutive positive integers ending in i. In this way, we obtain a tableaux $T=T({\varpi })$ of shape $\tau ({\varpi })$ with strictly decreasing rows. Define the depth of a box B of T to be the distance from B to the end of the row it occupies. Form a reduced decomposition for a permutation ${\omega }_T$ by listing the entries in the boxes of T in decreasing order of depth, with the entries of a fixed depth listed in increasing order. It then follows from the definition of ${\omega }$ that ${\omega }_T={\omega }$ .

Example 6. Let ${\varpi }:=(1,3,6,7,9,4,8,2,5)$ be the vexillary permutation in $S_9$ with code $\gamma =(0,1,3,3,4,1,2,0,0)$ . The initial indices are $1$ , $2$ , $3$ , $4$ , and $6$ with associated indices $7$ , $7$ , $5$ , $5$ , and $7$ , respectively. The reduced decomposition for the canonical permutation ${\omega }$ is

$$\begin{align*}s_4s_3s_4s_6s_2s_3s_4s_5s_6s_1s_2s_3s_4s_5s_6 \end{align*}$$

and we have ${\varpi } {\omega }=(9,7,6,8,4,3,1,2,5)$ , with code $(8,6,5,5,3,2,0,0,0)$ . We also have $\lambda =(4,3,3,2,1,1)$ , $\widehat {\gamma }=(0,2,5,6,8,3,5,0)$ , and $\widehat {\lambda }=(8,6,5,5,3,2)$ . The tableau $T({\varpi })$ on the skew diagram $\tau ({\varpi })$ is displayed in Figure 1.

Figure 1 The tableau T on the skew diagram $\widehat {\lambda }/\lambda $ .

It would be interesting to find analogs of the canonical permutation ${\omega }$ and the tableaux $T({\varpi })$ for the amenable elements in the other classical Lie types.

4.2. Type A degeneracy loci

Let $E\to {\mathfrak X}$ be a vector bundle of rank n on a complex algebraic variety ${\mathfrak X}$ , assumed to be smooth for simplicity. Let ${\varpi }\in S_n$ be amenable of shape $\lambda $ , and let ${\mathfrak f}$ and ${\mathfrak g}$ be the left and right flags of ${\varpi }$ , respectively. Consider two complete flags of subbundles of E

$$\begin{align*}0 \subset E_1\subset \cdots \subset E_n=E \ \ \, \mathrm{and} \, \ \ 0 \subset F_1\subset \cdots \subset F_n=E \end{align*}$$

with ${\mathrm {rank}} E_r={\mathrm {rank}} F_r=r$ for each r. Define the degeneracy locus ${\mathfrak X}_{\varpi }\subset {\mathfrak X}$ as the locus of $x \in {\mathfrak X}$ such that

$$\begin{align*}\dim(E_r(x)\cap F_s(x))\geq \#\,\{\,i \leq r \ |\ {\varpi}_i> n-s\,\} \ \, \forall \, r,s. \end{align*}$$

Assume further that ${\mathfrak X}_{\varpi }$ has pure codimension $\ell ({\varpi })$ in ${\mathfrak X}$ . The next result, which follows from Theorem 1 and Fulton’s work [Reference Fulton8], will be a formula for the cohomology class $[{\mathfrak X}_{\varpi }]$ in ${\mathrm {H}}^{2\ell ({\varpi })}({\mathfrak X})$ in terms of the Chern classes of the bundles $E_r$ and $F_s$ . Recall that for any integer p, the class $c_p(E-E_r-F_s)$ is defined by the equation

$$\begin{align*}c(E-E_r-F_s) := c(E)c(E_r)^{-1}c(F_s)^{-1} \end{align*}$$

of total Chern classes.

Theorem 3 [Reference Fulton8]

For any amenable permutation ${\varpi }\in S_n$ , we have

(18) $$ \begin{align} [{\mathfrak X}_{\varpi}] = s_{\lambda'}(E-E_{{\mathfrak f}}-F_{n-{\mathfrak g}}) = R^\emptyset\, c_{\lambda}(E-E_{{\mathfrak f}}-F_{n-{\mathfrak g}}) \end{align} $$

in the cohomology ring ${\mathrm {H}}^*({\mathfrak X})$ .

The Chern polynomial in (18) is interpreted as the image of the Schur polynomial $s_{\lambda '}(\mathfrak {c}) :=R^\emptyset \mathfrak {c}_{\lambda }$ under the ${\mathbb Z}$ -linear map which sends the noncommutative monomial $\mathfrak {c}_\alpha $ to $\prod _j c_{\alpha _j}(E-E_{{\mathfrak f}_j}-F_{n-{\mathfrak g}_j})$ , for every integer sequence $\alpha $ .

5.1. Definitions and main theorem

Let w be a signed permutation with A-code $\gamma $ and shape $\lambda =\mu +\nu $ , with $\ell =\ell (\lambda )$ and $m=\ell (\mu )$ . Choose $k\geq 0$ , and assume that w is increasing up to k. If $k=0$ , this condition is vacuous, while if $k\geq 1,$ it means that $0<w_1<\cdots <w_k$ . Eventually, k will be the first right descent of w, but the increased flexibility is useful.

List the entries $w_{k+1},\ldots ,w_n$ in increasing order:

$$\begin{align*}u_1<\cdots < u_m < 0 < u_{m+1} < \cdots < u_{n-k}. \end{align*}$$

Define a sequence $\beta (w)$ by

$$\begin{align*}\beta(w):=(u_1+1,\ldots,u_m+1,u_{m+1},\ldots,u_{n-k}). \end{align*}$$

and the denominator set $D(w)$ by

(19) $$ \begin{align} D(w):=\{(i,j) \ |\ 1\leq i<j \leq n-k \ \, \text{and} \ \, u_i+ u_j < 0 \}. \end{align} $$

This notation suppresses the dependence of $\beta (w)$ and $D(w)$ on k. Observe that the inequality $u_i+u_j<0$ in (19) is equivalent to $\beta _i(w)+ \beta _j(w) \leq 0$ .

Definition 6. Suppose that $w\in W_n$ has code $\gamma =\gamma (w)$ and $k\geq 0$ . The k-truncated A-code ${}^k\gamma ={}^k\gamma (w)$ is defined by

$$\begin{align*}{}^k\gamma(w):=(\gamma_{k+1}, \gamma_{k+2},\ldots, \gamma_n). \end{align*}$$

If k is the first right descent of w, then we call ${}^k\gamma (w)$ the truncated A-code of w. We let $\xi =\xi (w)$ be the conjugate of the partition whose parts are the non-zero entries of ${}^k\gamma (w)$ arranged in weakly decreasing order.

Clearly, an element $w\in W_n$ increasing up to k with a given k-truncated A-code C is uniquely determined by the set of elements $\{w_{k+1},\ldots ,w_n\}$ , or equivalently, by the sequence $\beta (w)$ .

Let $ v(w)$ be the unique k-Grassmannian element obtained by reordering the entries $w_{k+1},\ldots ,w_n$ to be increasing. For example, if $w:=(2,4,7,5,8,\overline {3},1,\overline {6})$ and $k:=3$ , then $v(w)=(2,4,7,\overline {6},\overline {3},1,5,8)$ . Note that the map $w \mapsto v(w)$ is a bijection from the set of elements in $W_n$ increasing up to k with k-truncated A-code C onto the set of k-Grassmannian elements in $W_n$ , such that $\beta (v(w))=\beta (w)$ and

$$\begin{align*}\ell(v(w)) = \ell(w)-\sum_{i=k+1}^n C_i = \ell(w)-\sum_{j=1}^{n-k} \gamma_{k+j}. \end{align*}$$

In particular, if $w,\overline {w}$ are two such elements, then $\ell (w)>\ell (\overline {w})$ if and only if $\ell (v(w))>\ell (v(\overline {w}))$ .

For any three integer vectors $\alpha ,\beta ,\rho \in {\mathbb Z}^\ell $ , define ${}^{\rho }c^\beta _\alpha :={}^{\rho _1}c^{\beta _1}_{\alpha _1}\,{}^{\rho _2}c^{\beta _2}_{\alpha _2}\dots $ . Given any raising operator $R=\prod _{i<j}R_{ij}^{n_{ij}}$ , let $R\, {}^{\rho }c^\beta _{\alpha } := {}^{\rho }c^\beta _{R\alpha }$ .

Proposition 5. Fix an integer $k\geq 0$ . Suppose that w and $\overline {w}$ are elements in $W_n$ increasing up to k with the same k-truncated A-code C, such that $\ell (w)=\ell (\overline {w})+1$ and $s_iv(w)=v(\overline {w})$ for some simple reflection $s_i$ . Assume that we have

$$\begin{align*}{\mathfrak C}_w = R^{D(w)}\,{}^{{\kappa}}c^{\beta(w)}_{\lambda(w)} \end{align*}$$

for some integer sequence ${\kappa }$ . Then, we have

$$\begin{align*}{\mathfrak C}_{\overline{w}} = R^{D(\overline{w})}\,{}^{{\kappa}}c^{\beta(\overline{w})}_{\lambda(\overline{w})}. \end{align*}$$

Proof. Set $F_w:=R^{D(w)} \, {}^{{\kappa }}c^{\beta (w)}_{\lambda (w)}$ , so we know that ${\mathfrak C}_w=F_w$ . As equation (8) gives $\partial ^y_i{\mathfrak C}_w = {\mathfrak C}_{\overline {w}}$ , it will suffice to show that $\partial ^y_i F_w = F_{\overline {w}}$ . The proof of this will follow the argument of [Reference Tamvakis and Wilson31, Proposition 5].

Let $\mu :=\mu (w)$ , $\nu :=\nu (w)$ , $\lambda :=\lambda (w)=\mu +\nu $ , $\overline {\mu }:=\mu (\overline {w})$ , $\overline {\nu }:=\nu (\overline {w})$ , $\overline {\lambda }:=\lambda (\overline {w})= \overline {\mu }+\overline {\nu }$ , $\beta =\beta (w)$ , and $\overline {\beta }=\beta (\overline {w})$ . There are four possible cases for w, discussed below. In each case, we have $\overline {\lambda }\subset \lambda $ , so that $\overline {\lambda }_p=\lambda _p-1$ for some $p\geq 1$ and $\overline {\lambda }_j=\lambda _j$ for all $j\neq p$ .

(a) $v(w) = (\cdots \overline {1} \cdots )$ with $i=0$ . In this case, we have $D(w)=D(\overline {w})$ . Since clearly $\nu =\overline {\nu }$ and $\overline {\mu }_p=\mu _p-1$ for $p=\ell (\mu )$ , while $\overline {\mu }_j=\mu _j$ for all $j\neq p$ , it follows that $\beta _p=i$ , $\overline {\beta }_p= i + 1$ , while $\beta _j=\overline {\beta }_j$ for all $j \neq p$ .

(b) $v(w) = (\cdots i \cdots \overline {i+1} \cdots )$ . In this case $D(w)=D(\overline {w})$ , and we have $\nu =\overline {\nu }$ and $\overline {\mu }_p=\mu _p-1$ , while $\overline {\mu }_j=\mu _j$ for all $j\neq p$ . It follows that $\beta _p = -i$ , $\overline {\beta }_p=-i+1$ , and $\beta _j=\overline {\beta }_j$ for all $j \neq p$ .

(c) $v(w) = (\cdots \overline {i+1} \cdots i \cdots )$ . In this case $D(w)=D(\overline {w})\cup \{(p,q)\}$ , where $\beta _p=-i$ and $\beta _q=i$ . We see similarly that $\overline {\beta }_p=-i+1$ and $\overline {\beta }_q=i+1$ , while $\beta _j=\overline {\beta }_j$ for all $j \notin \{p,q\}$ .

(d) $v(w) = (\cdots i+1 \cdots i \cdots )$ . In this case, we have $D(w)=D(\overline {w})$ while clearly $\mu =\overline {\mu }$ . We deduce that $\overline {\nu }_p=\nu _p-1$ , while $\overline {\nu }_j=\nu _j$ for all $j\neq p$ . We must show that $\beta _p=i$ , and hence $\overline {\beta }_p=i+1$ , and $\beta _j=\overline {\beta }_j$ for all $j\neq p$ .

Note that if $w_r=i+1$ , then $r\in [1,k]$ . Since $w_j\geq w_r$ for all $j\in [r,k]$ , and the sequence $\beta (w)$ is strictly increasing, we deduce that $\beta _g=i$ exactly when $g=\gamma _r(w)$ . We have $\gamma _r(\overline {w})=\gamma _r(w)-1=g-1$ , while $\gamma _j(w)=\gamma _j(\overline {w})$ for $j\neq r$ . It follows that $\overline {\nu }_g=\nu _g-1$ , while $\overline {\nu }_j=\nu _j$ for all $j\neq g$ . In other words, $g=p$ , as desired.

To simplify the notation, set $c^{\rho }_{\alpha }:={}^{{\kappa }}c^{\rho }_\alpha $ , for any integer sequences $\alpha $ and $\rho $ . In cases (a), (b), or (d), it follows using the left Leibnitz rule and Lemma 3(a) that for any integer sequence $\alpha =(\alpha _1,\ldots ,\alpha _\ell )$ , we have

$$ \begin{align*} \partial^y_i c^{\beta}_\alpha &= c^{(\beta_1,\ldots,\beta_{p-1})}_{(\alpha_1,\ldots,\alpha_{p-1})} \left(\partial^y_i(c^{\beta_p}_{\alpha_p})\,c^{(\beta_{p+1},\ldots,\beta_\ell)}_{(\alpha_{p+1},\ldots,\alpha_\ell)} +s^y_i(c^{\beta_p}_{\alpha_p})\,\partial^y_i (c^{(\beta_{p+1},\ldots,\beta_\ell)}_{(\alpha_{p+1},\ldots,\alpha_\ell)})\right) \\ &=c^{(\beta_1,\ldots,\beta_{p-1})}_{(\alpha_1,\ldots,\alpha_{p-1})} \left(c^{\beta_p+1}_{\alpha_p-1}\,c^{(\beta_{p+1},\ldots,\beta_\ell)}_{(\alpha_{p+1},\ldots,\alpha_\ell)} +s^y_i(c^{\beta_p}_{\alpha_p})\cdot 0\right) =c^{(\beta_1,\ldots,\beta_p+1,\ldots,\beta_\ell)}_{(\alpha_1,\ldots,\alpha_p-1,\ldots,\alpha_\ell)} = c^{\overline{\beta}}_{\alpha-\epsilon_p}. \end{align*} $$

Since $\lambda -\epsilon _p=\overline {\lambda }$ , we deduce that if R is any raising operator, then

$$\begin{align*}\partial^y_i (R \,c^{\beta}_{\lambda}) = \partial^y_i (c^{\beta}_{R\lambda}) = c^{\overline{\beta}}_{R\lambda-\epsilon_p} = R \,c^{\overline{\beta}}_{\overline{\lambda}}. \end{align*}$$

As $R^{D(w)}= R^{D(\overline {w})}$ , we conclude that

$$\begin{align*}\partial^y_i(F_w) = \partial^y_i (R^{D(w)} c^{\beta}_{\lambda}) = R^{D(\overline{w})} c^{\overline{\beta}}_{\overline{\lambda}} = F_{\overline{w}}. \end{align*}$$

In case (c), it follows from the left Leibnitz rule as in the proof of Lemma 3(b) that for any integer sequence $\alpha =(\alpha _1,\ldots ,\alpha _\ell )$ , we have

$$ \begin{align*} \partial^y_i c^{\beta}_\alpha &= \partial^y_i c^{(\beta_1,\ldots,-i,\ldots,i,\ldots,\beta_{\ell})}_{(\alpha_1,\ldots,\alpha_p,\ldots,\alpha_q, \ldots,\alpha_\ell)} \\ &= c^{(\beta_1,\ldots,-i+1,\ldots,i+1,\ldots,\beta_{\ell})}_{(\alpha_1,\ldots,\alpha_p-1,\ldots,\alpha_q,\ldots,\alpha_\ell)} +c^{(\beta_1,\ldots,-i+1,\ldots,i+1,\ldots,\beta_{\ell})}_{(\alpha_1,\ldots,\alpha_p,\ldots,\alpha_q-1,\ldots,\alpha_\ell)} = c^{\overline{\beta}}_{\alpha-\epsilon_p} + c^{\overline{\beta}}_{\alpha-\epsilon_q}. \end{align*} $$

Since $\lambda -\epsilon _p=\overline {\lambda }$ , we deduce that if R is any raising operator, then

$$\begin{align*}\partial^y_i (R \,c^{\beta}_\lambda) = \partial^y_i (c^{\beta}_{R\lambda}) = c^{\overline{\beta}}_{R\lambda-\epsilon_p} + c^{\overline{\beta}}_{R\lambda-\epsilon_q} = R \,c^{\overline{\beta}}_{\overline{\lambda}} + RR_{pq}\,c^{\overline{\beta}}_{\overline{\lambda}}. \end{align*}$$

As $R^{D(w)}+R^{D(w)} R_{pq} = R^{D(\overline {w})}$ , we conclude that

$$\begin{align*}\partial^y_i(F_w) = \partial^y_i(R^{D(w)} c^{\beta}_\lambda) = R^{D(w)} c^{\overline{\beta}}_{\overline{\lambda}} +R^{D(w)} R_{pq}c^{\overline{\beta}}_{\overline{\lambda}} = R^{D(\overline{w})} c^{\overline{\beta}}_{\overline{\lambda}} = F_{\overline{w}}. \end{align*}$$

Proposition 6. Suppose that $w\in W_n$ is such that $\gamma (w)$ is a partition. Then, we have

(20) $$ \begin{align} {\mathfrak C}_w = R^{D(w)} \, {}^{\nu(w)}c^{\beta(w)}_{\lambda(w)}. \end{align} $$

Proof. Assume first that $w_i<0$ for each i, and $\gamma (w)$ is a partition. We claim that

(21) $$ \begin{align} {\mathfrak C}_w = R^{\infty} \, {}^{\nu(w)}c^{-\delta_{n-1}}_{\lambda(w)}. \end{align} $$

The proof of (21) is by descending induction on $\ell (w)$ . One knows from [Reference Ikeda, Mihalcea and Naruse13, Theorem 1.2] and [Reference Tamvakis29, Proposition 3.2] that (21) is true for the longest element $w_0$ in $W_n$ , since $\nu (w_0)=\delta _{n-1}$ and $\lambda (w_0)=\delta _n+\delta _{n-1}$ .

Suppose that $w\neq w_0$ is such that $\gamma (w)$ is a partition, and the shape of w equals $\delta _n+\nu $ . Then, $\nu \subset \delta _{n-1}$ and $\nu \neq \delta _{n-1}$ . Let $r\geq 1$ be the largest integer such that $\nu _i=n-i$ for $i\in [1,r]$ , and let $j:=\nu _{r+1}+1\leq n-r-1$ . Then, $ws_j$ is of length $\ell (w)+1$ and satisfies the same conditions, $\nu (ws_j)=\nu (w)+\epsilon _{r+1}$ , and $\lambda (ws_j)=\lambda (w)+\epsilon _{r+1}$ . Using Lemma 3(a), for any integer sequence $\alpha =(\alpha _1,\ldots ,\alpha _n)$ , we have

$$\begin{align*}\partial^x_j\left({}^{\nu(ws_j)}c^{-\delta_{n-1}}_{\alpha}\right) = {}^{\nu(w)}c^{-\delta_{n-1}}_{\alpha-\epsilon_{r+1}}. \end{align*}$$

We deduce that

$$\begin{align*}{\mathfrak C}_w = \partial^x_j({\mathfrak C}_{ws_j}) = \partial^x_j\left(R^{\infty} \, {}^{\nu(ws_j)}c^{-\delta_{n-1}}_{\lambda(ws_j)}\right)= R^{\infty} \, {}^{\nu(w)}c^{-\delta_{n-1}}_{\lambda(w)}, \end{align*}$$

proving the claim. Equation (20) now follows, by combining (21) with the $k=0$ case of Proposition 5.

Corollary 1. Suppose that $w\in W_n$ is increasing up to k and the k-truncated A-code ${}^k\gamma $ is a partition. Let $k^{n-k}+\xi (w)=(k+\xi _1,\ldots ,k+\xi _{n-k})$ . Then, we have

(22) $$ \begin{align} {\mathfrak C}_w = R^{D(w)} \, {}^{k^{n-k}+\xi(w)}c^{\beta(w)}_{\lambda(w)}. \end{align} $$

Definition 7. Let $k\geq 0$ denote the first right descent of $w\in W_n$ . List the entries $w_{k+1},\ldots ,w_n$ in increasing order:

$$\begin{align*}u_1<\cdots < u_m < 0 < u_{m+1} < \cdots < u_{n-k}. \end{align*}$$

We say that a simple transposition $s_i$ for $i\geq 1$ is w-negative (respectively, w-positive) if $\{i,i+1\}$ is a subset of $\{-u_1,\ldots ,-u_m\}$ (respectively, of $\{u_{m+1}\ldots ,u_{n-k}\}$ ). Let $\sigma ^-$ (respectively, $\sigma ^+$ ) be the longest subword of $s_{n-1}\dots s_1$ (respectively, of $s_1\dots s_{n-1}$ ) consisting of w-negative (respectively, w-positive) simple transpositions. A modification of $w\in W_n$ is an element ${\omega } w$ , where ${\omega }\in S_n$ is such that $\ell ({\omega } w)=\ell (w)-\ell ({\omega })$ , and ${\omega }$ has a reduced decomposition of the form $R_1\dots R_{n-1}$ , where each $R_j$ is a (possibly empty) subword of $\sigma ^-\sigma ^+$ and all simple reflections in $R_p$ are also contained in $R_{p+1}$ , for each $p<n-1$ .

Remark 3. (a) The integer vector $\alpha =(\alpha _1,\ldots ,\alpha _p)$ is called unimodal if for some $j\in [1,p]$ , we have

$$\begin{align*}\alpha_1\leq \alpha_2 \leq \cdots \leq \alpha_j\geq \alpha_{j+1} \geq \cdots \geq \alpha_p. \end{align*}$$

The element $w\in W_n$ is leading if and only if the A-code of the extended sequence $(0,w_1,w_2,\ldots ,w_n)$ , where we have set $w(0):=0$ , is unimodal.

(b) Given an element $w\in W_n$ , there is an easy algorithm to decide whether or not w is amenable. One simply applies all possible inverse modifications to w and checks if any of these result in a leading element.

Example 7. Consider the leading element $w:=(2,4,6,5,\overline {1},\overline {3})$ in $W_6$ , with $k=3$ , $\gamma (w)= (2,2,3,2,1,0)$ , $\mu (w)=(3,1)$ , $\nu (w)=(5,4,1)$ , $\xi (w)=(2,1)$ , and $\lambda (w)=(8,5,1)$ . We have $\beta (w)=(-2,0,5)$ and $D(w)=\{(1,2)\}$ , so Corollary 1 gives

$$\begin{align*}{\mathfrak C}_w = R^{\{12\}} \, {}^{(5,4,3)}c_{(8,5,1)}^{(-2,0,5)} = \frac{1-R_{12}}{1+R_{12}}(1-R_{13})(1-R_{23})\, {}^{(5,4,3)}c_{(8,5,1)}^{(-2,0,5)}. \end{align*}$$

In the following, we will assume that w has first right descent at $k\geq 0$ and $\xi $ is as in Definition 6. Let $\psi :=(\gamma _k,\ldots ,\gamma _1)$ , $\phi :=\psi '$ , $\ell :=\ell (\lambda ),$ and $m:=\ell (\mu )$ . We then have

(23) $$ \begin{align} \lambda = \phi+\xi+\mu \end{align} $$

and $\lambda _1>\cdots > \lambda _m > \lambda _{m+1}\geq \cdots \geq \lambda _{\ell }$ .