2.1. Pattern group

Let $\Delta _n=\{(i,j)\mid 1\le i< j\le n\}.$ A closed subset ${\mathcal{D}}\subset \Delta _n$ is a subset with the property that ${\mathcal{D}}$ contains $(i,k)$ whenever it contains both $(i,j)$ and $(j,k).$ For any closed subset ${\mathcal {D}}\subset \Delta _n,$ the pattern group $G_{{\mathcal{D}}}(k)$ is

$$ \begin{align*}G_{{\mathcal {D}}}(k):=\{1+a_{ij}e_{i,j}\mid (i,j)\in{\mathcal {D}},\ a_{ij}\in k\}.\end{align*} $$

It is clear that

$$ \begin{align*}|G_{{\mathcal {D}}}({\mathbb {F}}_q)|=q^{|{\mathcal{D}}|}.\end{align*} $$

Let ${\mathrm {Lie}}(G_{{\mathcal {D}}}(k))$ be the Lie algebra of $G_{{\mathcal {D}}}(k)$ . We identify the dual space of ${\mathrm {Lie}}(G_{{\mathcal {D}}}(k))$ with ${\mathrm {Lie}}(G_{{\mathcal {D}}}(k))^t$ via the trace map

$$ \begin{align*}{\mathrm{tr}}:{\mathrm{Mat}}_{n}(k)\times{\mathrm{Mat}}_{n}(k)\to k,\quad (A,B)\mapsto {\mathrm{tr}}(A\cdot B).\end{align*} $$

2.2. Mixed ladder determinantal variety

We review some notation about mixed ladder determinantal varieties; for the details, see [Reference Gonciulea and Miller4, Reference Gorla5].

Let $X=(X_{ba})$ , $1\le b\le m$ , $1\le a\le n$ , be an $m\times n$ matrix, whose entries $X_{ba}$ are indeterminates. Given $1= b_{1}<\cdots < b_{h}<m$ , $1<a_{1}<\cdots <a_{h}= n$ , let

$$ \begin{align*}L=\{X_{ba}\mid \text{there exists } 1\le i\le h\ \text{such that } b_{i}\le b\le m,1\le a\le a_{i} \}\end{align*} $$

be a one-sided ladder in X. Set

$$ \begin{align*}L_{i}:=\{X_{ba}\mid b_{i}\le b\le m, 1\le a\le a_{i} \};\end{align*} $$

then $L=\bigcup _{i=1}^{h}L_{i}$ . Here, we also regard L and $L_i$ as submatrices of X.

Let $\bar {k}[L]$ be the polynomial ring $\bar {k}[X_{ba},X_{ba}\in L]$ and $\mathbb {A}(L):=\mathbb {A}^{|L|}$ be the associated affine space. Fix $\underline {\mathbf {t}}=(t_{1},\ldots ,t_{h})\in \mathbb {Z}_{+}^{h}$ . Let $I_i$ be the ideal of $\bar {k}[L]$ generated by the $t_{i}$ -minors of X contained in $L_{i}$ . Let $DV_{\underline {\mathbf {t}}}(L)\subset \mathbb {A}(L)$ be the variety defined by the ideal

$$ \begin{align*}I_{\underline{\mathbf{t}}}(L):=(I_1,\ldots,I_h)=\sum_{i=1}^{h}I_{i}.\end{align*} $$

We assume that the size of $L_{i}$ is at least $t_{i}\times t_{i}$ , that is, $1\le t_{i}\le \min \{m-b_{i}+1,a_{i}\}$ . We also assume $I_i\nsubseteq I_j\ \text {for } i\neq j$ , and then we call $DV_{\underline {\mathbf {t}}}(L)$ a one-sided mixed determinantal variety associated with L and $\underline {\mathbf {t}}$ . If $L=L_i$ and $\underline {\mathbf {t}}=(t_{i})$ , then $DV_{\underline {\mathbf {t}}}(L)$ becomes the determinantal variety. We simply denote it by $\mathrm {D}_{m,n}^{l}$ , where $L_i$ is of size $m\times n$ and $t_i=l$ . It is clear that $DV_{\underline {\mathbf {t}}}(L)$ and $D^{l}_{m,n}$ can be defined over an arbitrary field.

2.3. Schubert variety and Bruhat cell

We give some preliminaries about Schubert varieties and Bruhat cells. For details, we refer to [Reference Borel1, Reference Gonciulea and Miller4, Reference Humphreys8]. Let G be a simply connected simple algebraic group split over k. Let $T\subset G$ be a split maximal torus and B be a Borel subgroup of G defined over k that contains T. Let $W=N_{G}(T)/T$ be the Weyl group of G. Let S be the set of simple roots relative to $(B,T)$ . For a subset $S_Q\subset S$ , denote by Q the standard parabolic subgroup associated to $S_Q$ . Let $W_{Q}$ be the Weyl group of Q, which is a parabolic subgroup of W. Let $W^{Q}$ be the set of minimal length representatives of $W/W_{Q}$ . For $w\in W$ , we denote the coset $wQ$ in $G/Q$ by $e_{w,Q}$ . We have Bruhat decomposition

$$ \begin{align*}G/Q=\bigsqcup\limits_{w\in W^{Q}}Be_{w,Q}.\end{align*} $$

Each $Be_{w,Q}$ is called a Bruhat cell, which is isomorphic to an affine space. Let $X_{Q}(w)$ be the Zariski closure of $Be_{w,Q}$ in $G/Q$ . Then, $X_{Q}(w)$ with the canonical reduced structure is called the Schubert variety. It is well known that the Schubert variety $X_{Q}(w)$ can be written as $X_{Q}(w)=\bigsqcup _{u\in W^{Q}, u\le w}Be_{u,Q}$ , where $\le $ is the Bruhat order. Let $B^{-}$ be the Borel subgroup of G opposite to B. We have opposite decomposition

$$ \begin{align*}G/Q=\bigsqcup\limits_{w\in W^{Q}}B^{-}e_{w,Q}.\end{align*} $$

Here, $B^{-}e_{w,Q}$ is called an opposite Bruhat cell, which is also an affine space. Consider the canonical projection $\pi :G/B\to G/Q$ ; we have the following properties:

1. (1) $\pi ^{-1}(B^{-}e_{w,Q})= \bigsqcup_{u\in wW_{Q}}B^{-}e_{u,B}$ ;

2. (2) for $w\in W^{Q}$ , the restriction of $\pi $ to $Be_{w,B}$ is an isomorphism onto $Be_{w,Q}$ .

For $u,v\in W$ , denote $X_{u}^{v}:=Be_{u,B}\cap B^{-}e_{v,B}$ to be the intersection of $Be_{u,B}$ and $B^{-}e_{v,B}$ ; for $u,v\in W^{Q}$ , define $Y_{u}^{v}:=Be_{u,Q}\cap B^{-}e_{v,Q}$ to be the intersection of $Be_{u,Q}$ and $B^{-}e_{v,Q}$ . Here, $X_{u}^{v}$ and $Y_{u}^{v}$ are nonempty if and only if $v\le u$ . All varieties above can be defined over an arbitrary field.

4.1. Clifford theory and coadjoint orbits

Let us review Clifford theory and coadjoint orbit theory [Reference Kirillov12, Reference Serre16]. Assume that the finite group $G =A\rtimes H$ is a semidirect product of A by H, where A is an abelian normal subgroup of G. Let $\widehat {A}$ be the set of all irreducible characters of A. Let $h\in H$ act on $\widehat A$ by $h\cdot \chi =\chi ^h,$ where $\chi ^h\in \widehat A$ is given by

$$ \begin{align*}\chi^h(a)=\chi(h^{-1}ah) \quad\text{for } a\in A.\end{align*} $$

Let $ H_\chi =\{h\in H\ |\ \chi ^h=\chi \}$ be the stabilizer of $\chi $ in H. For $\chi \in \widehat A$ , we identify it with a character of $A\rtimes H_{\chi }$ by $\chi (ah):=\chi (a)$ , where $a\in A$ and $h\in H_{\chi }$ . For $\tau \in \widehat H_\chi $ , we identify it with a character of $A\rtimes H_{\chi }$ via the canonical projection $A\rtimes H_{\chi }\to H_{\chi }$ . Then, the irreducible characters of G can be described in terms of induced characters of the following form.

Fix a nontrivial one-dimensional character $\psi $ of k. Note that when $A=1+{\mathrm {Lie}}(A)$ is an abelian pattern group with subalgebra ${\mathrm {Lie}}(A)\subset {\mathrm {Lie}}({\mathrm {U}}_n(k))$ ; then

$$ \begin{align*}\widehat A=\{\psi_T\ | \ T\in {\mathrm{Lie}}(A)^t\},\end{align*} $$

where

$$ \begin{align*}\psi_T(x):=\psi({\mathrm{tr}} Tx)\quad\text{for } x\in A.\end{align*} $$

Definition 4.2. For a matrix $c=(c_{i,j})\in {\mathrm {Mat}}_n(k)$ , denote

$$ \begin{align*} {\mathrm{Supp}}(c):=\{(i, j)\in \Delta_n \mid c_{i,j}\ne 0\}.\end{align*} $$

For a subset $X\subset {\mathrm {Mat}}_n(k)$ , denote

$$ \begin{align*} {\mathrm{Supp}}(X):=\bigcup_{c\in X}{\mathrm{Supp}}(c). \end{align*} $$

Let $\mathscr {P}$ be a pattern group such that $\mathscr {P}=A\rtimes H=(1+{\mathrm {Lie}}(A))\rtimes H$ with abelian pattern subgroups A and pattern subgroup H. We may identify the dual space of ${\mathrm {Lie}}(A)$ with ${\mathrm {Lie}}(A)^t$ via the trace map. The coadjoint action of $g\in H$ on $\alpha \in {\mathrm {Lie}}(A)^t$ in [Reference Kirillov12] is given by

$$ \begin{align*} (g\cdot \alpha)(a)={\mathrm{tr}}(\alpha g^{-1}ag) \quad\text{for } a\in {\mathrm{Lie}}(A). \end{align*} $$

It can be identified with $\alpha \mapsto [g\alpha g^{-1}]_{A}$ , where $[\cdot ]_{A}$ is the projection from ${\mathrm {Mat}}_n(k)$ to ${\mathrm {Lie}}(A)^t.$ More precisely, for $m=(m_{i,j})\in {\mathrm {Mat}}_n(k)$ ,

$$ \begin{align*} ([m]_{A})_{i,j}=m_{i,j} \text{ if } (j,i)\in {\mathrm{Supp}}({\mathrm{Lie}}(A)) \quad\text{and}\quad ([m]_{A})_{i,j}=0 \text{ if } (j,i)\notin {\mathrm{Supp}}({\mathrm{Lie}}(A)). \end{align*} $$

Define the action of H on ${\mathrm {Lie}}(A)^t$ by

$$ \begin{align*} g\circ_A \alpha:=[g\alpha g^{-1}]_{A}\quad\text{for } g\in H\text{ and } \alpha\in {\mathrm{Lie}}(A)^t.\end{align*} $$

Since $g\cdot \alpha (a)={\mathrm {tr}}([g\alpha g^{-1}]_{A}a)$ for all $a\in {\mathrm {Lie}}(A)$ , the action $\circ _A$ of H on ${\mathrm {Lie}}(A)^t$ coincides with the coadjoint action, whose orbits are called coadjoint orbits.

4.2. Pattern subgroups in $\mathrm {U}_{a,b,c,d}$

Let

$$ \begin{align*}\mathrm{U}_{a,b,c,d}({\mathbb {F}}_q):=\left\{\begin{pmatrix} \mathrm{Id}_{a}&A&B&C\\ 0&\mathrm{Id}_{b}&D&E\\ 0&0&\mathrm{Id}_{c}&F\\ 0&0&0&\mathrm{Id}_{d} \end{pmatrix}\kern3pt\middle|\kern-3pt \begin{array}{c@{}}A\in{\mathrm{Mat}}_{a\times b}({\mathbb {F}}_q),B\in{\mathrm{Mat}}_{a\times c}({\mathbb {F}}_q),\\ C\in{\mathrm{Mat}}_{a\times d}({\mathbb {F}}_q), D\in{\mathrm{Mat}}_{b\times c}({\mathbb {F}}_q),\\ E\in{\mathrm{Mat}}_{b\times d}({\mathbb {F}}_q),F\in{\mathrm{Mat}}_{c\times d}({\mathbb {F}}_q) \end{array}\right\}\end{align*} $$

be the unipotent radical of a four-block standard parabolic subgroup of $\mathrm {GL}_{n}({\mathbb {F}}_q)$ , where $a+b+c+d=n$ . Let

$$ \begin{align*}\mathrm{U}_{{\mathcal {H}}}({\mathbb {F}}_q):=\left\{\begin{pmatrix} \mathrm{Id}_{a}&A&0&0\\ 0&\mathrm{Id}_{b}&0&0\\ 0&0&\mathrm{Id}_{c}&F\\ 0&0&0&\mathrm{Id}_{d} \end{pmatrix}\kern3pt\middle|\kern3pt A\in{\mathrm{Mat}}_{a\times b}({\mathbb {F}}_q),F\in{\mathrm{Mat}}_{c\times d}({\mathbb {F}}_q)\right\}\end{align*} $$

and

$$ \begin{align*}\mathrm{U}_{{\mathcal {A}}}({\mathbb {F}}_q):=\left\{\begin{pmatrix} \mathrm{Id}_{a}&0&B&C\\ 0&\mathrm{Id}_{b}&D&E\\ 0&0&\mathrm{Id}_{c}&0\\ 0&0&0&\mathrm{Id}_{d} \end{pmatrix}\kern3pt\middle|\kern-3pt \begin{array}{c@{}}B\in{\mathrm{Mat}}_{a\times c}({\mathbb {F}}_q),C\in{\mathrm{Mat}}_{a\times d}({\mathbb {F}}_q) \\ D\in{\mathrm{Mat}}_{b\times c}({\mathbb {F}}_q),E\in{\mathrm{Mat}}_{b\times d}({\mathbb {F}}_q) \end{array}\right\}\end{align*} $$

be two abelian pattern subgroups of $\mathrm {U}_{a,b,c,d}({\mathbb {F}}_q)$ . We have semidirect product decomposition:

$$ \begin{align*}\mathrm{U}_{a,b,c,d}({\mathbb {F}}_q)\simeq\mathrm{U}_{{\mathcal {A}}}({\mathbb {F}}_q)\rtimes\mathrm{U}_{{\mathcal {H}}}({\mathbb {F}}_q).\end{align*} $$

Let $\mathrm {G}$ be a pattern subgroup in $\mathrm {U}_{a,b,c,d}({\mathbb {F}}_q)$ . Then, $\mathrm {G}_{{\mathcal {H}}}:=\mathrm {G}\cap \mathrm {U}_{{\mathcal {H}}}({\mathbb {F}}_q)$ and $\mathrm {G}_{{\mathcal {A}}}:=\mathrm {G}\cap \mathrm {U}_{{\mathcal {A}}}({\mathbb {F}}_q)$ are abelian pattern subgroups of $\mathrm {G}$ , and we also have semidirect product decomposition:

$$ \begin{align*}\mathrm{G}\simeq\mathrm{G}_{{\mathcal {A}}}\rtimes\mathrm{G}_{{\mathcal {H}}}.\end{align*} $$

4.3. Character-number formula of $\mathrm {\mathbf {G}}$

We simply write $U(X,Y,Z,W)$ for

$$ \begin{align*}\begin{pmatrix} 0_{a\times a}&0&0&0\\ 0&0_{b\times b}&0&0\\ X&Z&0_{c\times c}&0\\ Y&W&0&0_{d\times d} \end{pmatrix}.\end{align*} $$

Let

$$ \begin{align*}V(\mathbf{h}^1,\mathbf{h}^2):=\begin{pmatrix} \mathrm{Id}_{a}&\mathbf{h}^{1}&0&0\\ 0&\mathrm{Id}_{b}&0&0\\ 0&0&\mathrm{Id}_{c}&\mathbf{h}^{2}\\ 0&0&0&\mathrm{Id}_{d} \end{pmatrix}\in\mathrm{G}_{\mathcal{H}},\end{align*} $$

where $\mathbf {h}^1=(\mathbf {h}^{1}_{i,j})_{a\times b}$ and $\mathbf {h}^2=(\mathbf {h}^{2}_{i,j})_{c\times d}$ are two matrices that take the values in $\mathbb {F}_q$ and satisfy $\mathbf {h}^k_{i,j}=0\ (k=1,2)$ if $(i,j)\notin {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{\mathcal {H}}))$ . Let $U(X,Y,Z,W)\in {\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^{t}$ and consider the coadjoint action of $V(\mathbf {h}^1,\mathbf {h}^2)$ on $U(X,Y,Z,W)$ . To be more precise, that is,

$$ \begin{align*} &[V(\mathbf{h}^1,\mathbf{h}^2)\cdot U(X,Y,Z,W)\cdot V(-\mathbf{h}^1,-\mathbf{h}^2)]_{\mathrm{G}_{{\mathcal {A}}}}\\ &\quad=U(X,Y,Z,W)+[U(\mathbf{h}^{2}Y,0,\mathbf{h}^{2} W-X\mathbf{h}^1-\mathbf{h}^{2}Y\mathbf{h}^{1},-Y \mathbf{h}^{1})]_{\mathrm{G}_{{\mathcal{A}}}}. \end{align*} $$

Now, we want to determine the stabilizer of $U(X,Y,Z,W)$ in $\mathrm {G}_{\mathcal {H}}$ . To do this, we consider the following equation:

(4-1) $$ \begin{align} [U(\mathbf{h}^{2}Y,0,\mathbf{h}^{2} W-X\mathbf{h}^1-\mathbf{h}^{2}Y\mathbf{h}^{1},-Y \mathbf{h}^{1})]_{\mathrm{G}_{{\mathcal {A}}}}=0. \end{align} $$

It is clear that (4-1) gives algebraic equations in variables $\mathbf {h}^k_{i,j}$ ( $k=1,2$ ), where $(i,j)\in {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{\mathcal {H}})).$ Then, the solutions of (4-1) give the stabilizer of $U(X,Y,Z,W)$ in $\mathrm {G}_{\mathcal {H}}$ under the coadjoint action, that is,

$$ \begin{align*}\{V(\mathbf{h}^1,\mathbf{h}^2)\mid (\mathbf{h}^1,\mathbf{h}^2)\ \text{is a solution of}\ (4\text{-}1)\}.\end{align*} $$

Also, we consider another equation as follows:

(4-2) $$ \begin{align} [U(\mathbf{h}^{2}Y,0,\mathbf{h}^{2} W-X\mathbf{h}^1,-Y \mathbf{h}^{1})]_{\mathrm{G}_{{\mathcal{A}}}}=0. \end{align} $$

Equation (4-2) forms a system of linear equations in the same variables as (4-1).

Proof. Note that the equation $[U(A,B,C,D)]_{\mathrm {G}_{\mathcal {A}}}=0$ equals the system of equations

$$ \begin{align*}\begin{cases} {[U(A,0,0,0)]_{\mathrm{G}_{{\mathcal{A}}}}=0},\\ {[U(0,B,0,0)]_{\mathrm{G}_{{\mathcal {A}}}}=0},\\{[U(0,0,C,0)]_{\mathrm{G}_{{\mathcal {A}}}}}=0,\\{[U(0,0,0,D)]_{\mathrm{G}_{{\mathcal {A}}}}}=0, \end{cases}\kern-10pt\end{align*} $$

due to the linearity of the projection $[-]_{\mathrm {G}_{\mathcal {A}}}$ . It is enough to show that

(4-3) $$ \begin{align} [U(\mathbf{h}^{2}Y,0,\mathbf{h}^{2} W-X\mathbf{h}^1-\mathbf{h}^{2}Y\mathbf{h}^{1},0)]_{\mathrm{G}_{{\mathcal {A}}}}=0 \end{align} $$

and

(4-4) $$ \begin{align} [U(\mathbf{h}^{2}Y,0,\mathbf{h}^{2} W-X\mathbf{h}^1,0)]_{\mathrm{G}_{{\mathcal {A}}}}=0 \end{align} $$

share the same solutions over ${\mathbb {F}}_q$ . We have

$$ \begin{align*}({4\text{-}3})\Leftrightarrow\begin{cases} {[U(\mathbf{h}^{2}Y,0,0,0)]_{\mathrm{G}_{{\mathcal {A}}}}=0},\\ {[U(0,0,\mathbf{h}^{2} W-X\mathbf{h}^1-\mathbf{h}^{2}Y\mathbf{h}^{1},0)]_{\mathrm{G}_{{\mathcal {A}}}}=0}, \end{cases}\end{align*} $$

and

$$ \begin{align*}({4\text{-}4})\Leftrightarrow\begin{cases} {[U(\mathbf{h}^{2}Y,0,0,0)]_{\mathrm{G}_{{\mathcal {A}}}}=0},\\ {[U(0,0,\mathbf{h}^{2} W-X\mathbf{h}^1,0)]_{\mathrm{G}_{{\mathcal {A}}}}=0}. \end{cases}\kern-10pt\end{align*} $$

A single equation in $[U(0,0,\mathbf {h}^{2} W-X\mathbf {h}^1-\mathbf {h}^{2}Y\mathbf {h}^{1},0)]_{\mathrm {G}_{{\mathcal {A}}}}=0$ , which is indexed by $(i,l)\in {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t)$ , is formed by

$$ \begin{align*} \sum\limits_{j}\mathbf{h}^{2}_{i,j}w_{j,l}-\sum\limits_{j} x_{i,j}\mathbf{h}^{1}_{j,l}-\sum\limits_{k,(k,l)\in{\mathrm{Supp}}({\mathrm{Lie}}(\mathrm{G}_{{\mathcal {H}}}))}\bigg(\sum\limits_{j}\mathbf{h}^{2}_{i,j}y_{j,k}\bigg)\mathbf{h}^{1}_{k,l}=0. \end{align*} $$

Now, we show that $\sum \limits _{j}\mathbf {h}^{2}_{i,j}y_{j,k}=0\ ((k,l)\in {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{\mathcal {H}})))$ appears as a single equation in $[U(\mathbf {h}^2Y,0,0,0)]_{\mathrm {G}_{\mathcal {A}}}=0$ . It is enough to show $(i,k)\in {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{\mathcal {A}})^t)$ . Note that

$$ \begin{align*}(i,l)\in{\mathrm{Supp}}({\mathrm{Lie}}(\mathrm{G}_{{\mathcal{A}}})^t)\quad \text{implies}\quad \kern-1pt(l,i)\in{\mathrm{Supp}}({\mathrm{Lie}}(\mathrm{G}_{{\mathcal{A}}}).\end{align*} $$

Additionally, we also have $(k,l)\kern1.4pt{\in}\kern1.4pt {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{{\mathcal {H}}}))$ , which implies $(k,i)\kern1.4pt{\in}\kern1.4pt {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}}))$ since $\mathrm {G}_{{\mathcal{A}}}$ is normal. Hence, $(i,k)\in {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{{\mathcal{A}}})^t).$ Let ${\mathbb {F}}_q[\mathbf {h}^k_{i,j}]$ be the polynomial ring with indeterminates $\mathbf {h}^k_{i,j}$ over ${\mathbb {F}}_q$ , where $(i,j)\in {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{\mathcal {H}}))$ . Now, consider the two ideals of ${\mathbb {F}}_q[\mathbf {h}^k_{i,j}]$ determined by (4-3) and (4-4). The arguments above show that these two ideals are actually the same, which finishes the proof.

We denote the coefficient matrix of (4-2) by ${\mathscr {C}}(X,Y,Z,W)$ under a fixed total order of $\mathbf {h}^{k}_{i,j}$ ( $k=1,2$ ) and observe that ${\mathscr {C}}(X,Y,Z,W)$ does not depend on Z. Define

$$ \begin{align*}\mathfrak{C}(\mathrm{G}):=\{{\mathscr {C}}(X,Y,Z,W)\mid U(X,Y,Z,W)\in{\mathrm{Lie}}(\mathrm{G}_{{\mathcal{A}}})^t\}.\end{align*} $$

Let $\pi _c: {\mathrm {Lie}}(\mathrm {G}_{{\mathcal{A}}})^t\to \mathfrak {C}(\mathrm {G})$ be the canonical map via $U(X,Y,Z,W)\mapsto {\mathscr {C}}(X,Y,Z,W)$ . For $e\in {\mathbb {N}}$ , define

$$ \begin{align*}\mathfrak{C}(\mathrm{G},e):=\{{\mathscr {C}}(X,Y,Z,W)\in\mathfrak{C}(\mathrm{G})\mid {\mathrm{rank}}({\mathscr {C}}(X,Y,Z,W))=e\}.\end{align*} $$

Then, $\mathfrak {C}(\mathrm {G})=\bigcup _{e=0}^{\infty }\mathfrak {C}(\mathrm {G},e)$ is a finite partition of $\mathfrak {C}(\mathrm {G})$ . Denote the fibre of $\pi _c$ over $\mathfrak {C}(\mathrm {G}, e)$ by $\tilde {\mathfrak {C}}(\mathrm {G}, e)$ .

Proposition 4.4 (Characters number formula).

Let $\mathrm {G}$ be a pattern subgroup of $\mathrm {U}_{a,b,c,d}({\mathbb {F}}_q)$ . Then, the number of irreducible characters of degree $q^e$ is

$$ \begin{align*}q^{-2e}\cdot|\tilde{\mathfrak{C}}(\mathrm{G}, e)|\cdot|\mathrm{G}_{{\mathcal{H}}}|.\end{align*} $$

Furthermore, the set of integers that occur as degrees of irreducible characters of $\mathrm {G}$ is precisely $\{q^{e}\mid \mathfrak {C}(\mathrm {G},e)\neq \emptyset \}$ .

Proof. Fix a nontrivial one-dimensional character $\psi $ of ${\mathbb {F}}_{q}$ . Then we have that $\hat {\mathrm {G}}_{{\mathcal{A}}}=\{\psi _{a}\mid a\in {\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t\}$ , where

$$ \begin{align*}\psi_{a}(x):=\psi({\mathrm{tr}}\ ax)\quad \text{for } x\in \mathrm{G}_{{\mathcal{A}}},\end{align*} $$

which identify $\hat {\mathrm {G}}_{{\mathcal{A}}}$ with ${\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t$ via $\Psi :{\mathrm {Lie}}(\mathrm {G}_{\mathcal {A}})^t\to \hat {\mathrm {G}}_{{\mathcal {A}}}, a\mapsto \psi _{a}$ . Moreover, $\Psi $ is compatible with the action of $\mathrm {G}_{{\mathcal {H}}}$ on $\hat {\mathrm {G}}_{{\mathcal{A}}}$ and the coadjoint action of $\mathrm {G}_{{\mathcal {H}}}$ on ${\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t$ . To be more precise, we have a commutative diagram as follows:

Below, we use Theorem 4.1 to construct the irreducible characters of $\mathrm {G}$ . Under the identification of $\hat {\mathrm {G}}_{{\mathcal {A}}}$ and ${\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t$ , we no longer distinguish between the coadjoint orbit $\mathbb {O}_a$ in ${\mathrm {Lie}}(\mathrm {G}_{\mathcal {A}})^t$ and its corresponding orbit $\mathbb {O}_{\psi _a}$ in $\hat {\mathrm {G}}_{{\mathcal {A}}}$ , as well as the stabilizer subgroups of $a\in {\mathrm {Lie}}(\mathrm {G}_{\mathcal {A}})^t$ and $\psi _a\in \hat {\mathrm {G}}_{{\mathcal {A}}}$ . For a coadjoint orbit $\mathbb {O}_{a}$ with $a\in {\mathrm {Lie}}(\mathrm {G}_{\mathcal {A}})^t$ , let ${\mathrm {G}_{{\mathcal {H}}}}_a$ be the stabilizer of a in $\mathrm {G}_{{\mathcal {H}}}$ . Clearly, ${\mathrm {G}_{{\mathcal {H}}}}_a$ is an abelian group (but usually not a pattern group) and $\hat {\mathrm {G}}_{{\mathcal {H}} a}$ consists of $|{\mathrm {G}_{{\mathcal {H}}}}_a|$ one-dimensional irreducible characters. Let $\tau $ be an irreducible character of $\mathrm {G}_{\mathcal {H}a}$ , which is one-dimensional. Theorem 4.1(1) tells us, $\mathrm {Ind}^{\mathrm {G}}_{\mathrm {G}_{\mathcal {A}}\rtimes \mathrm {G}_{\mathcal {H}a}}\psi _a\otimes \tau $ is an irreducible character of $\mathrm {G}$ of degree $[\mathrm {G}_{\mathcal {H}}:\mathrm {G}_{\mathcal {H}a}]$ , where $[\mathrm {G}_{{\mathcal {H}}}:{\mathrm {G}_{{\mathcal{H}}}}_a]$ is the index of ${\mathrm {G}_{{\mathcal {H}}}}_a$ in $\mathrm {G}_{{\mathcal {H}}}$ . Theorem 4.1(2) tells us that we can construct $|\mathrm {G}_{\mathcal {H}a}|$ pairwise nonisomorphic irreducible characters associated with $\mathbb {O}_a$ , that is, $\{\mathrm {Ind}^{\mathrm {G}}_{\mathrm {G}_{\mathcal {A}}\rtimes \mathrm {G}_{\mathcal {H}a}}\psi _a\otimes \tau \mid \tau \in \hat {\mathrm {G}}_{\mathcal {H}a}\}$ . By Theorem 4.1(3), we can construct all of the irreducible characters of $\mathrm {G}$ if we consider all of the coadjoint orbits and the stabilizers.

Now, we calculate the number of degree $q^e$ irreducible characters. We write

$$ \begin{align*}h=1+\begin{pmatrix} 0_{a\times a}&h^1&0&0\\ 0&0_{b\times b}&0&0\\ 0&0&0_{c\times c}&h^2\\ 0&0&0&0_{d\times d} \end{pmatrix}\in\mathrm{G}_{{\mathcal {H}}},\end{align*} $$

where $h^k=(h^{k}_{i,j})$ ( $k=1,2$ ), $h^{k}_{i,j}=0$ if $(i,j)\notin {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {G}_{{\mathcal {H}}}))$ . By Lemma 4.3,

$$ \begin{align*}{\mathrm{G}_{{\mathcal {H}}}}_a=\{h\in\mathrm{G}_{{\mathcal{H}}}\mid (h^{k}_{i,j})\ \text{is a solution of}\ (4\text{-}2),k=1,2\}.\end{align*} $$

We write $a=U(X,Y,Z,W)$ ; then $|{\mathrm {G}_{{\mathcal{H}}}}_a|=q^{-{\mathrm {rank}}({\mathscr {C}}(X,Y,Z,W))}\cdot |\mathrm {G}_{{\mathfrak {H}}}|$ , and the degree of the irreducible characters associated to ${\mathbb {O}}_{a}$ is $q^{{\mathrm {rank}}({\mathscr {C}}(X,Y,Z,W))}$ . We denote the number of $\mathrm {G}_{{\mathcal {H}}}$ -coadjoint orbits with stabilizer (of a representative element) of index $q^{e}$ by $N_{e}$ , then the number of irreducible characters of $\mathrm {G}$ of degree $q^{e}$ is

$$ \begin{align*}N_{e}\cdot q^{-e}\cdot |\mathrm{G}_{{\mathcal {H}}}|.\end{align*} $$

Let

$$ \begin{align*}{\mathfrak {O}}_{e}:=\bigcup\limits_{[\mathrm{G}_{{\mathcal {H}}}:{\mathrm{G}_{{\mathcal {H}}}}_a]=q^{e}}{\mathbb {O}}_{a}\end{align*} $$

be the set of unions of all coadjoint orbits whose stabilizer in $\mathrm {G}_{{\mathcal{H}}}$ has index $q^{e}$ . Then,

$$ \begin{align*}{\mathfrak {O}}_{e}=\{U(X,Y,Z,W)\in{\mathrm{Lie}}(\mathrm{G}_{{\mathcal{A}}})^t\mid {\mathrm{rank}}({\mathscr {C}}(X,Y,Z,W))=e\},\end{align*} $$

which is just the set $\tilde {\mathfrak {C}}(\mathrm {G},e)$ . Since the cardinality of a $\mathrm {G}_{{\mathcal{H}}}$ -orbit equals the index of its stabilizer, the coadjoint orbits in ${\mathfrak {O}}_{e}$ have the same cardinality, which is $q^{e}$ . Hence, $N_{e}=q^{-e}\cdot |\tilde {\mathfrak {C}}(\mathrm {G},e)|$ and the number of irreducible characters of $\mathrm {G}$ of degree $q^e$ is

$$ \begin{align*}q^{-2e}\cdot |\tilde{\mathfrak{C}}(\mathrm{G},e)|\cdot |\mathrm{G}_{{\mathcal{H}}}|.\end{align*} $$

The remaining part is clear, since the map $\pi _c$ is surjective and all the irreducible characters can be given in the above way.

Example 4.7. Let $\mathrm {G}=\mathrm {U}_{0,b,c,d}({\mathbb {F}}_q)$ be the unipotent radical of a 3-block parabolic subgroup of $\mathrm {GL}_{n}({\mathbb {F}}_q)$ . In this case, each $a\in {\mathrm {Lie}}(\mathrm {G}_{\mathcal {A}})^t$ is of the form $U(0,0,Z,W)$ and

$$ \begin{align*}{\mathscr {C}}(0,0,Z,W)=\mathrm{diag}(\overbrace{W,\ldots,W,\ldots,W}^{c \text{ times}}).\end{align*} $$

Then, ${\mathrm {rank}}({\mathscr {C}}(0,0,Z,W))=c\cdot {\mathrm {rank}}(W)$ . Let $m_k$ be the number of matrices of rank k in ${\mathrm {Mat}}_{d\times b}({\mathbb {F}}_q)$ . We have $|\tilde {\mathfrak {C}}(\mathrm {G},ck)|=q^{cb} m_k$ and the number of degree $q^{ck}$ irreducible characters is

$$ \begin{align*}q^{(b+d-2k)c}m_k,\end{align*} $$

which gives another proof of the character number part of [Reference Nien15, Theorem 2.3].

5.1. Polynomial property for pattern groups

We say a pattern group $G_{{\mathcal {D}}}({\mathbb {F}}_q)$ has the polynomial property if the number of its characters of degree $q^e$ is a polynomial in q with integral coefficients for any $e\in {\mathbb {N}}$ .

By Proposition 4.4, we reduce the problem of the polynomial property of $\mathrm {G}$ to the polynomial property of $\tilde {\mathfrak {C}}(\mathrm {G},e)$ . In other words, $\mathrm {G}$ has the polynomial property if and only if $|\tilde {\mathfrak {C}}(\mathrm {G},e)|$ is a polynomial in q with integral coefficients.

Example 5.1. Let us focus on the case $\mathrm {G}={\mathrm {U}}_{0,b,c,d}({\mathbb {F}}_q)$ again. We have $|\tilde {\mathfrak {C}}(\mathrm {G},e)|\neq 0$ only if $c\mid e$ . For $e=ck$ ,

$$ \begin{align*}\tilde{\mathfrak{C}}(\mathrm{G},ck)=q^{(b+d-2k)c}|\mathfrak{C}(\mathrm{G},ck)|.\end{align*} $$

We can identify $\mathfrak {C}(\mathrm {G},ck)$ with $\{T\in {\mathrm {Mat}}_{d\times b}({\mathbb {F}}_q)\mid {\mathrm {rank}}(T)=k\}$ . The latter can be viewed as $\mathrm {D}_{d,b}^{k+1}({\mathbb {F}}_q)\backslash \mathrm {D}_{d,b}^{k}({\mathbb {F}}_q)$ . The polynomial property of $\mathrm {G}$ is a consequence of the fact that the determinantal varieties are polynomial-counting.

Let

$$ \begin{align*}{\mathcal {D}}_{m}=\{(i,j)\in\Delta_n \mid \ j-i \ge m \}.\end{align*} $$

We have

$$ \begin{align*}{\mathrm {U}}_n({\mathbb {F}}_q)=G_{{\mathcal{D}}_1}({\mathbb {F}}_q)\ge G_{{\mathcal {D}}_2}({\mathbb {F}}_q)\ge\cdots\ge G_{{\mathcal {D}}_m}({\mathbb {F}}_q)\ge\cdots\ge G_{{\mathcal{D}}_n}({\mathbb {F}}_q)=1,\end{align*} $$

which becomes the lower central series of ${\mathrm {U}}_n({\mathbb {F}}_q)$ .

We assume $({n}/{3})\le m<({n}/{2})$ ( $n\ge 5$ ). Under this assumption, $G_{{\mathcal {D}}_m}({\mathbb {F}}_q)$ is not abelian and is a pattern subgroup of some ${\mathrm {U}}_{0,b,c,d}({\mathbb {F}}_q)$ .

Proof. It is enough to show ${\mathrm {Supp}}({\mathrm {Lie}}(G_{\mathcal {D}_m}({\mathbb {F}}_q)))\subset {\mathrm {Supp}}({\mathrm {Lie}}(\mathrm {U}_{0,m,n-2m,m}({\mathbb {F}}_q))).$ Indeed,

$$ \begin{align*} &{\mathrm{Supp}}({\mathrm{Lie}}(\mathrm{U}_{0,m,n-2m,m}({\mathbb {F}}_q)))\\&\quad=\{(i,j)\mid 1\le i\le n-m,\ m+1\le j\le n\}\backslash\{(i,j)\mid m+1\le i,j\le n-m\} \end{align*} $$

and ${\mathrm {Supp}}({\mathrm {Lie}}(G_{\mathcal {D}_m}({\mathbb {F}}_q)))=\{(i,j)\in \Delta _n\mid j-i\ge m\}.$ One can verify the inclusion easily, which completes the proof.

Set $s:=n-2m$ . Denote by ${\mathrm {Tri}}_s({\mathbb {F}}_q)$ the set of all lower triangular matrices of size $s\times s$ over ${\mathbb {F}}_q$ . Let $\mathbf {a}=(a_{i,j})_{1\le j\le i\le s}\in {\mathrm {Tri}}_{s}({\mathbb {F}}_q)$ and define

which is a block matrix with i th block

It is clear that f is an injective map from ${\mathrm {Tri}}_{s}({\mathbb {F}}_q)$ to ${\mathrm {Tri}}_{(1/2)s(s+1)}({\mathbb {F}}_q)$ .

Example 5.3. For $s=3$ , $\mathbf {a}=\begin {pmatrix} a_{1,1}&&\\ a_{2,1}&a_{2,2}&\\ a_{3,1}&a_{3,2}&a_{3,3}\\ \end {pmatrix}$ , we have

$$ \begin{align*}f(\mathbf{a})=\begin{pmatrix} a_{1,1}&&&&&\\ a_{2,1}&&&&&\\ a_{3,1}&&&&&\\ &a_{2,1}&a_{2,2}&&&\\ &a_{3,1}&a_{3,2}&&&\\ &&&a_{3,1}&a_{3,2}&a_{3,3}\\ \end{pmatrix},\end{align*} $$

which has three blocks.

Let $\mathrm {G}=G_{{\mathcal {D}}_m}({\mathbb {F}}_q)$ ( $({n}/{3})\le m<({n}/{2}),\ n\ge 5$ ), which is a pattern subgroup in ${\mathrm {U}}_{0,m,n-2m,m}({\mathbb {F}}_q)$ by Lemma 5.2. For $U(0,0,Z,W)\in {\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t$ , $W=(w_{i,j})_{m\times m},$ define

to be the lower triangular matrix of size $s\times s$ , which is in the left corner of W.

Denote the number of degree $q^e$ irreducible characters of $G_{{\mathcal{D}}_m}({\mathbb {F}}_q)$ by $N_{G_{{\mathcal{D}}_{m}}({\mathbb {F}}_q),\ e}.$

Theorem 5.4. For $({n}/{3})\le m<({n}/{2})$ with $n\ge 5$ and $e\in {\mathbb {N}}$ ,

$$ \begin{align*}N_{G_{{\mathcal {D}}_{m}}({\mathbb {F}}_q),\ e}=q^{{(s+m)(s+m+1)}/{2}-{s(s+1)}/{2}-2e}\cdot|\mathfrak{C}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q),e)|.\end{align*} $$

Moreover, the set of integers that occur as degrees of irreducible characters of $G_{{\mathcal {D}}_{m}}({\mathbb {F}}_{q})$ is precisely $\{q^e\mid 0\le e\le r_{\max }(s),\ e\in {\mathbb {N}}\}$ , where $s=n-2m$ and

$$ \begin{align*}r_{\max}(s):=\begin{cases} \dfrac{s^2+2s}{4}&\text{if } n \text{ is even,}\\[4pt] \dfrac{(s+1)^2}{4}&\text{if } n \text{ is odd.} \end{cases}\end{align*} $$

Proof. By Proposition 4.4,

$$ \begin{align*}N_{G_{{\mathcal{D}}_{m}}({\mathbb {F}}_q),\ e}=q^{{s(s+1)}/{2}-2e}\cdot|\tilde{\mathfrak{C}}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q),e)|.\end{align*} $$

For $U(0,0,Z,W)\in {\mathrm {Lie}}(\mathrm {G}_{{\mathcal {A}}})^t$ ,

$$ \begin{align*}{\mathscr {C}}(0,0,Z,W)=f({\mathscr {M}}(0,0,Z,W)).\end{align*} $$

Noting that f is injective, the map $\pi _c$ in this case is a fibration with the cardinality of the fibre equal to $q^{{(s+m)(s+m+1)}/{2}-s(s+1)}$ . We have

$$ \begin{align*}|\tilde{\mathfrak{C}}(G_{{\mathcal{D}}_m}({\mathbb {F}}_q),e)|=q^{{(s+m)(s+m+1)}/{2}-s(s+1)}\cdot|\mathfrak{C}(G_{{\mathcal{D}}_m}({\mathbb {F}}_q),e)|\end{align*} $$

and hence,

$$ \begin{align*}N_{G_{{\mathcal {D}}_{m}}({\mathbb {F}}_q),\ e}=q^{{(s+m)(s+m+1)}/{2}-{s(s+1)}/{2}-2e}\cdot|\mathfrak{C}(G_{{\mathcal{D}}_m}({\mathbb {F}}_q),e)|.\end{align*} $$

For the remaining part, it is enough to prove the claim as follows. For any $n\in \mathbb {Z}_{+}$ and $0\le e\le r_{\max }(n)$ , there exists a $\mathbf {t}\in {\mathrm {Tri}}_{n}({\mathbb {F}}_q)$ such that ${\mathrm {rank}}(f(\mathbf {t}))=e$ . Furthermore, ${\mathrm {rank}}(f(\mathbf {t}))\le r_{\max }(n)$ for any $\mathbf {t}\in {\mathrm {Tri}}_{n}({\mathbb {F}}_q)$ .

We prove our claim by induction on n. For $n=1$ , it is clear. Now, we assume it holds for $n=n_{0} (n_{0}\ge 1)$ . By induction, for $0\le e\le r_{\max }(n_{0})$ , we can find

$$ \begin{align*} \mathbf{t}=\begin{pmatrix}t_{1,1}&&\\\vdots&\ddots&\\t_{n_{0},1}&\dots&t_{n_{0},n_{0}}\\\end{pmatrix}\in {\mathrm{Tri}}_{n_{0}}({\mathbb {F}}_q), \end{align*} $$

such that ${\mathrm {rank}}(f(\mathbf {t}))=e$ . Let

$$ \begin{align*} \mathbf{t}'=\begin{pmatrix}t_{1,1}&&&\\\vdots&\ddots&&\\t_{n_{0},1}&\dots&t_{n_{0},n_{0}}&\\0&\cdots&\cdots&0\end{pmatrix}\in{\mathrm{Tri}}_{n_{0}+1}({\mathbb {F}}_q). \end{align*} $$

We have . Consider the following matrices:

$$ \begin{align*} \tilde{\mathbf{t}}_{n_{0}+1,m}=\begin{pmatrix}\tilde{t}_{1,1}&&&\\\vdots&\ddots&&\\\tilde{t}_{n_{0},1}&\dots&\tilde{t}_{n_{0},n_{0}}&\\\delta_{m,n_{0}+1}&\dots&\dots&\delta_{m,1}\\\end{pmatrix}\in{\mathrm{Tri}}_{n_{0}+1}({\mathbb {F}}_q),\quad 1\le m\le \bigg[\frac{n_{0}+2}{2}\bigg], \end{align*} $$

where $\tilde {t}_{i,j}=1$ for $0\le i-j \le n_{0}-[({n_{0}+1})/{2}]\ \mathrm{and}\ \tilde {t}_{i,j}=0$ whenever $i-j\ge n_{0}-[({n_{0}+1})/ {2}]+1$ . One can verify that ${\mathrm {rank}}(f(\tilde {\mathbf {t}}_{n_{0}+1,m}))=r_{\max }(n_{0})+m$ for $1\le m\le [({n_{0}\kern1.4pt{+}\kern1.4pt2})/{2}]$ . In particular, ${\mathrm {rank}}(f(\tilde {\mathbf {t}}_{n_{0}+ 1,[({n_{0}+2})/{2}]}))\kern1.4pt{=}\kern1.4pt r_{\max }(n_{0})\kern1.4pt{+}\kern1.4pt[({n_{0}\kern1.4pt{+}\kern1.4pt 2})/{2}]\kern1.4pt{=}\kern1.4pt r_{\max }(n_{0}\kern1.3pt{+}\kern1.3pt1)$ . Finally,

$$ \begin{align*}{\mathrm{rank}}(f(\mathbf{t}))\le \sum\limits_{i=1}^{n} \min\{i,n-i+1\}=r_{\max}(n)\end{align*} $$

$\text {for any}\ \mathbf {t}\in {\mathrm {Tri}}_{n}({\mathbb {F}}_q).$

5.2. A partition of $\mathfrak {C}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q),e)$

Let $\underline {\lambda }=(\lambda _{1},\ldots ,\lambda _{s})\in \mathbb {Z}^{s}$ . If we have $0\le \lambda _{i}\le \min \{i,s-i+1\} \text { for } i=1,\ldots ,s,$ we call $\underline {\lambda }$ admissible. For $e\in {\mathbb {N}}$ , define

$$ \begin{align*}\Lambda(s,e):=\bigg\{ \underline{\lambda}\mid \underline{\lambda}\ \text{is admissible with } \sum\limits_{i=1}^{s}\lambda_{i}=e \bigg\}.\end{align*} $$

For $\mathbf {t}\in {\mathrm {Tri}}_{s}({\mathbb {F}}_{q})$ , define $\mathbf {t}_{i}:=\left(\begin {smallmatrix}t_{i,1}&\dots &t_{i,i}\\ \vdots &\ddots &\vdots \\ t_{s,1}&\cdots &t_{s,i}\end {smallmatrix}\right)$ to be the i th block of $f(\mathbf {t})$ for $1\le i\le s$ . We call $\mathbf {t}_{i}$ the i th submatrix of $\mathbf {t}$ . Clearly,

$$ \begin{align*}{\mathrm{rank}}(f(\mathbf{t}))=\sum\limits_{i=1}^{s}{\mathrm{rank}}(\mathbf{t}_{i}).\end{align*} $$

For an admissible $\underline {\lambda }$ , we define

$$ \begin{align*} V_{q}(s,\underline{\lambda})&:=\{\mathbf{t}\in {\mathrm{Tri}}_{s}({\mathbb {F}}_{q})\mid {\mathrm{rank}}(\mathbf{t}_{i})=\lambda_{i},i=1,\ldots,s \},\\\overline{V}_{q}(s,\underline{\lambda})&:=\{\mathbf{t}\in {\mathrm{Tri}}_{s}({\mathbb {F}}_{q})\mid 0\le {\mathrm{rank}}(\mathbf{t}_{i})\le\lambda_{i},i=1,\ldots,s\}.\end{align*} $$

Since $\mathfrak {C}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q))=f({\mathrm {Tri}}_{s}({\mathbb {F}}_q))$ and f is injective, we identify $\mathfrak {C}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q))$ with ${\mathrm {Tri}}_{s}({\mathbb {F}}_{q})$ via f. Then,

(5-1) $$ \begin{align} \mathfrak{C}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q),e)=\bigsqcup\limits_{\underline{\lambda}\in \Lambda(n,e)}V_{q}(s,\underline{\lambda}).\end{align} $$

Proof. For an admissible $\underline {\lambda }$ and $1\le i_{1}<\cdots <i_{t}\le s$ , define

$$ \begin{align*}\underline{\lambda^{i_{1},\ldots,i_{t}}}:=(\lambda_{1}^{i_{1},\ldots,i_{t}},\ldots,\lambda_{s}^{i_{1},\ldots,i_{t}}),\end{align*} $$

where

$$ \begin{align*}\lambda_{j}^{i_{1},\ldots,i_{t}}=\begin{cases} \lambda_{j}-1,& j\in\{i_{1},\ldots,i_{t}\}.\\ \lambda_{j}& \text{otherwise}.\\ \end{cases}\end{align*} $$

Then,

(5-2) $$ \begin{align} \overline{V}_{q}(s,\underline{\lambda^{i_{1},\ldots,i_{t}}})=\bigcap\limits_{r\in\{ i_{1},\ldots,i_{t} \}}\overline{V}_{q}(s,\underline{\lambda^{r}}). \end{align} $$

If $\underline {\lambda ^{i_{1},\ldots ,i_{t}}}$ is not admissible, take $\overline {V}_{q}(s,\underline {\lambda ^{i_{1},\ldots ,i_{t}}})=\emptyset $ . Note that

$$ \begin{align*}\overline{V}_{q}(s,\underline{\lambda})\backslash \bigcup\limits_{k=1}^{s}\overline{V}_{q}(s,\underline{\lambda^{k}})=V_{q}(s,\underline{\lambda})\end{align*} $$

and $\overline {V}_{q}(s,\underline {\lambda ^{k}})\subset \overline {V}_{q}(s,\underline {\lambda }),(k=1,\ldots ,s)$ . We have

$$ \begin{align*}|V_{q}(s,\underline{\lambda})|=|\overline{V}_{q}(s,\underline{\lambda})|-\bigg|\bigcup\limits_{k=1}^{s}\overline{V}_{q}(s,\underline{\lambda^{k}})\bigg|.\end{align*} $$

By the principle of inclusion-exclusion and (5-2),

(5-3) $$ \begin{align} |V_{q}(s,\underline{\lambda})|=|\overline{V}_{q}(s,\underline{\lambda})|+\sum\limits_{t=1}^{s}(-1)^{t}\sum\limits_{1\le i_{1}<\cdots<i_{t}\le s}|\overline{V}_{q}(s,\underline{\lambda^{i_{1},\ldots,i_{s}}})|. \end{align} $$

Then, by (5-1) and (5-3),

$$ \begin{align*} |\mathfrak{C}(G_{{\mathcal {D}}_m}({\mathbb {F}}_q),e)|=\sum\limits_{\underline{\lambda}\in \Lambda(n,e)}|\overline{V}_{q}(s,\underline{\lambda})|+\sum\limits_{\underline{\lambda}\in \Lambda(n,e)}\sum\limits_{t=1}^{s}(-1)^{t}\sum\limits_{1\le i_{1}<\cdots<i_{t}\le s}|\overline{V}_{q}(s,\underline{\lambda^{i_{1},\ldots,i_{s}}})|, \end{align*} $$

which is a linear combination of $|\overline {V}_{q}(s,\underline {\lambda })|$ with integral coefficients for some admissible $\underline {\lambda }$ .

Proof. Let $T=(T_{ba})_{s\times s}$ be an $s\times s$ lower triangular matrix with indeterminates $T_{ba},1\le a\le b\le s$ . For $1\le i\le s$ , let $T_{i}$ be the i th submatrix of T. For i such that $\lambda _{i}<\min \{i,s-i+1\}$ , let $I_{i}$ be the ideal of $\overline {{\mathbb {F}}}_{q}[T] :=\overline {{\mathbb {F}}}_{q}[T_{ba},T_{ba}\in T]$ generated by all $(\lambda _{i}+1)\times (\lambda _{i}+1)$ minors of $T_{i}$ ; for $\lambda _{i}=\min \{i,s-i+1\}$ , we let $I_{i}=(0)$ . Let $\mathfrak {I}=\{I_{i}\neq 0\mid i=1,\ldots ,s\}$ . Then, $\mathfrak {I}$ becomes a poset if we take the inclusion relationship between ideals as the partial order. Let

$$ \begin{align*}M=\{i\mid I_{i}\ \textit{is maximal in}\ \mathfrak{I} \}=\{i_{1}<\cdots<i_{|M|}\}.\end{align*} $$

Since $\underline {\lambda }\neq (\min \{i,s-i+1\})_{i=1}^{s}$ , we have $M\neq \emptyset $ . Let $T'=\bigcup \limits _{i\in M}T_{i}$ and

$$ \begin{align*}\underline{\lambda}'=(\lambda_{i_{1}}+1,\ldots,\lambda_{i_{|M|}}+1).\end{align*} $$

We have a one-sided mixed ladder determinantal variety $DV_{\underline {\lambda }'}(T')$ . We claim that $\overline {V}_{q}(s,\underline {\lambda })$ equals the set of ${\mathbb {F}}_{q}$ -rational points of $DV_{\underline {\lambda }'}(T')\times {\mathbb {A}}^{|T\backslash T'|}$ .

Note that $DV_{\underline {\lambda }'}(T')\times {\mathbb {A}}^{|T\backslash T'|}\cong \text {Spec}\ \overline {{\mathbb {F}}}_{q}[T]/(I_{1},\ldots ,I_{s})$ . It remains to show that $\overline {V}_{q}(s,\underline {\lambda })$ equals the set of common zeros of $I_{1},\ldots ,I_{s}$ over ${\mathbb {F}}_q$ . We identify ${\mathrm {Tri}}_{s}({\mathbb {F}}_{q})$ with the set of ${\mathbb {F}}_{q}$ -valued points of $\text {Spec}\ {\mathbb {F}}_{q}[T]$ . Let $\mathbf {t}=(t_{i,j})_{s\times s}\in {\mathrm {Tri}}_{s}({\mathbb {F}}_{q})$ be a common zero of $I_{1},\ldots ,I_{s}$ , and let $\mathbf {t}_{i}$ be the i th submatrix of $\mathbf {t}$ , $i=1,\ldots ,s$ .

However, if $\lambda _{i}=\min \{i,s-i+1\}$ , since $\mathbf {t}_{i}$ is of size $i\times (s-i+1)$ , we have ${\mathrm {rank}}(\mathbf {t}_{i})\le \lambda _{i}$ ; if $\lambda _{i}<\min \{i,s-i+1\}$ , since $(t_{i,j})_{s\times s}$ is a zero of $I_i$ and $I_{i}$ is generated by the $(\lambda _{i}+1)\times (\lambda _{i}+1)$ minors of $T_{i}$ , then all $(\lambda _{i}+1)\times (\lambda _{i}+1)$ minors of $\mathbf {t}_{i}$ are equal to zero, which force that ${\mathrm {rank}}(\mathbf {t}_{i})\le \lambda _{i}$ . Hence, $DV_{\underline {\lambda }'}(T')\times {\mathbb {A}}^{|T\backslash T'|}({\mathbb {F}}_{q})\subset \overline {V}_{q}(s,\underline {\lambda })$ . However, letting $\mathbf {t}'=(t^{\prime }_{i,j})_{s\times s}\in \overline {V}_{q}(s,\underline {\lambda })$ , we show it is a common zero of $I_{1},\ldots ,I_{s}$ : for $I_{i}\neq 0$ , it is generated by the $(\lambda _{i}+1)\times (\lambda _{i}+1)$ minors of $T_{i}$ and ${\mathrm {rank}}(\mathbf {t}^{\prime }_{i})\le \lambda _{i}$ , so $(t^{\prime }_{i,j})_{s\times s}$ is a zero of $I_{i}$ ; for $I_{i}=0$ , it holds clearly. The proposition follows.

5.3. Lehrer conjecture for $G_{{\mathcal {D}}_{m}}$

Now, we can prove the analogue to Lehrer’s conjecture for finite pattern group $G_{{\mathcal {D}}_{m}}({\mathbb {F}}_{q})$ ( $({n}/{3})\le m<({n}/{2}),\ n\ge 5$ ).