2.1 Notation

Throughout the paper, fix a non-archimedean local field $F$ with ring of integers ${\mathcal{O}}$ and absolute value $|\cdot |$. In principle it should be possible to deal with the archimedean case as well with proper adjustments, but we do not consider this case here.

From § 3 onward, $F$ is assumed to be of characteristic $0$.

If $H$ is an algebraic group over $F$, we often also use $H$ to denote $H(F)$.

We will consider complex, smooth representations of finite length of the groups $\operatorname{GL}_{n}(F)$, $n\geqslant 0$. We denote the set of irreducible representations of $\operatorname{GL}_{n}(F)$ (up to equivalence) by $\operatorname{Irr}\operatorname{GL}_{n}$ and set $\operatorname{Irr}=\bigcup _{n\geqslant 0}\operatorname{Irr}\operatorname{GL}_{n}$. We write $\operatorname{Irr}\operatorname{GL}_{0}=\{\mathbf{1}\}$. (In contrast, the one-dimensional trivial character of $\operatorname{GL}_{1}$ will be denoted by $\mathbf{1}_{F^{\ast }}$.) The subset of supercuspidal (respectively, square-integrable, essentially square-integrable, tempered, generic) representations will be denoted by $\operatorname{Irr}_{\operatorname{cusp}}$ (respectively, $\operatorname{Irr}_{\operatorname{sqr}}$, $\operatorname{Irr}_{\operatorname{esqr}}$, $\operatorname{Irr}_{\operatorname{tmp}}$, $\operatorname{Irr}_{\operatorname{gen}}$). Thus,

$$\begin{eqnarray}\operatorname{Irr}_{\operatorname{cusp}}\subset \operatorname{Irr}_{\operatorname{sqr}}\subset \operatorname{Irr}_{\operatorname{esqr}}\quad \text{and}\quad \operatorname{Irr}_{\operatorname{esqr}},\operatorname{Irr}_{\operatorname{tmp}}\subset \operatorname{Irr}_{\operatorname{gen}}\!.\end{eqnarray}$$

By convention $\mathbf{1}\in \operatorname{Irr}_{\operatorname{tmp}}$ but $\mathbf{1}\notin \operatorname{Irr}_{\operatorname{esqr}}$.

Let $\unicode[STIX]{x1D70B}$ be a representation of $\operatorname{GL}_{n}(F)$. We denote by $\unicode[STIX]{x1D70B}^{\vee }$ the contragredient of $\unicode[STIX]{x1D70B}$ and by $\operatorname{soc}(\unicode[STIX]{x1D70B})$ the socle of $\unicode[STIX]{x1D70B}$ (the maximal semisimple subrepresentation of $\unicode[STIX]{x1D70B}$). If $\unicode[STIX]{x1D70B}$ is non-zero, then we write $\deg \unicode[STIX]{x1D70B}=n$, the degree of $\unicode[STIX]{x1D70B}$. For any character $\unicode[STIX]{x1D714}$ of $F^{\ast }$ (i.e. $\unicode[STIX]{x1D714}\in \operatorname{Irr}\operatorname{GL}_{1}$) we denote by $\unicode[STIX]{x1D70B}\unicode[STIX]{x1D714}$ the representation obtained from $\unicode[STIX]{x1D70B}$ by twisting by the character $\unicode[STIX]{x1D714}\circ \det$. For instance, $\unicode[STIX]{x1D70B}|\cdot |$ is the twist of $\unicode[STIX]{x1D70B}$ by $|\det |$. We also write $J_{P}(\unicode[STIX]{x1D70B})$ for the (normalized) Jacquet module of $\unicode[STIX]{x1D70B}$ with respect to a parabolic subgroup $P$ of $\operatorname{GL}_{n}$, defined over $F$. If $\unicode[STIX]{x1D70F}\in \operatorname{Irr}\operatorname{GL}_{n}$, then we write $\unicode[STIX]{x1D70F}\leqslant \unicode[STIX]{x1D70B}$ if $\unicode[STIX]{x1D70F}$ occurs as a subquotient of $\unicode[STIX]{x1D70B}$, that is, if $\unicode[STIX]{x1D70F}$ occurs in the Jordan–Hölder sequence of $\unicode[STIX]{x1D70B}$. If $\unicode[STIX]{x1D70F}$ occurs with multiplicity one in the Jordan–Hölder sequence of $\unicode[STIX]{x1D70B}$, then we write $\unicode[STIX]{x1D70F}\,{\leqslant}_{\operatorname{unq}}\unicode[STIX]{x1D70B}$.

If $\unicode[STIX]{x1D70B}_{1},\ldots ,\unicode[STIX]{x1D70B}_{k}$ are representations of $\operatorname{GL}_{n_{1}}(F),\ldots ,\operatorname{GL}_{n_{k}}(F)$ respectively, then we denote the representation parabolically induced from $\unicode[STIX]{x1D70B}_{1}\otimes \cdots \otimes \unicode[STIX]{x1D70B}_{k}$ (normalized induction), with respect to the standard parabolic subgroup of block upper triangular matrices, by $\unicode[STIX]{x1D70B}_{1}\times \cdots \times \unicode[STIX]{x1D70B}_{k}$ and refer to it as the product representation. We also use the notation $\operatorname{Ind}_{H}^{G}$ and $\operatorname{ind}_{H}^{G}$ to denote induction and induction with compact support (both normalized) from a subgroup $H$ of $G$.

For any $\unicode[STIX]{x1D70F}\in \operatorname{Irr}_{\operatorname{esqr}}$, let $\mathbf{e}(\unicode[STIX]{x1D70F})$ be the unique real number $s$ such that the twisted representation $\unicode[STIX]{x1D70F}|\cdot |^{-s}$ is unitarizable (i.e. has a unitary central character). Note that $\mathbf{e}(\unicode[STIX]{x1D70F}^{\vee })=-\mathbf{e}(\unicode[STIX]{x1D70F})$. Any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}$ can be written uniquely (up to permutation) as $\unicode[STIX]{x1D70B}=\unicode[STIX]{x1D70F}_{1}\times \cdots \times \unicode[STIX]{x1D70F}_{k}$ where $\unicode[STIX]{x1D70F}_{i}$ are essentially square-integrable. Let $\mathbf{e}(\unicode[STIX]{x1D70B})=\min \mathbf{e}(\unicode[STIX]{x1D70F}_{i})$. (For consistency we write $\mathbf{e}(\mathbf{1})=0$.) Then $\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\vee })\leqslant 0$ with equality if and only if $\unicode[STIX]{x1D70B}$ is essentially tempered. Moreover, $\unicode[STIX]{x1D70B}$ is tempered if and only if $\mathbf{e}(\unicode[STIX]{x1D70B})=\mathbf{e}(\unicode[STIX]{x1D70B}^{\vee })=0$. More generally, we will say that $\unicode[STIX]{x1D70B}$ is ‘approximately tempered’ (AT) if $\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\vee })+1>0$. Equivalently, $\mathbf{e}(\unicode[STIX]{x1D70F}_{i})-\mathbf{e}(\unicode[STIX]{x1D70F}_{j})<1$ for all $i,j$. It is known that every unitarizable $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}$ is (AT). (This follows from the classification of the unitary dual of $\operatorname{GL}_{n}(F)$ by Tadić [Reference TadićTad86].) We denote by $\operatorname{Irr}_{(AT)}$ the set of (AT) representations.

For any set $A$ we denote by ${\mathcal{M}}(A)$ the free commutative monoid generated by $A$, considered as an ordered monoid. Thus, an element of ${\mathcal{M}}(A)$ (a multiset of $A$) is a finite (possibly empty) formal sum of element of $A$.

2.2 Zelevinsky classification

We recall the well-known results and terminology of [Reference ZelevinskyZel80].

A segment $\unicode[STIX]{x1D6E5}$ (of length $l>0$ and center $\unicode[STIX]{x1D70C}\in \operatorname{Irr}_{\operatorname{cusp}}$) is a non-empty finite subset of $\operatorname{Irr}_{\operatorname{cusp}}$ of the form

$$\begin{eqnarray}\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(l)}=\{\unicode[STIX]{x1D70C}|\cdot |^{(1-l)/2},\unicode[STIX]{x1D70C}|\cdot |^{(3-l)/2},\ldots ,\unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2}\}.\end{eqnarray}$$

We define $\deg \unicode[STIX]{x1D6E5}=l\deg \unicode[STIX]{x1D70C}$ and write $e(\unicode[STIX]{x1D6E5})=\unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2}\in \operatorname{Irr}_{\operatorname{cusp}}$ (the endpoint of $\unicode[STIX]{x1D6E5}$),

$$\begin{eqnarray}\mathfrak{c}(\unicode[STIX]{x1D6E5})=\unicode[STIX]{x1D70C}|\cdot |^{(1-l)/2}+\unicode[STIX]{x1D70C}|\cdot |^{(3-l)/2}+\cdots +\unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2}\in {\mathcal{M}}(\operatorname{Irr}_{\operatorname{ cusp}})\end{eqnarray}$$

and $\unicode[STIX]{x1D6E5}^{\vee }=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}^{\vee }}^{(l)}$. For compatibility we also write $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(0)}=\emptyset$. Denote by ${\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$ the set of all segments. We extend $\deg$ additively to a function ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})\rightarrow \mathbb{Z}_{{\geqslant}0}$. Similarly, we extend $e$ and $\mathfrak{c}$ additively to functions ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})\rightarrow {\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$.

For any $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(l)}\in {\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$, let

$$\begin{eqnarray}Z(\unicode[STIX]{x1D6E5})=\operatorname{soc}(\unicode[STIX]{x1D70C}|\cdot |^{(1-l)/2}\times \unicode[STIX]{x1D70C}|\cdot |^{(3-l)/2}\times \cdots \times \unicode[STIX]{x1D70C}|\cdot |^{(l-1)/2})\in \operatorname{Irr}\operatorname{GL}_{ deg\unicode[STIX]{x1D6E5}}.\end{eqnarray}$$

(For compatibility we also set $Z(\emptyset )=\mathbf{1}$.) Then $Z(\unicode[STIX]{x1D6E5})^{\vee }=Z(\unicode[STIX]{x1D6E5}^{\vee })$. Given $\unicode[STIX]{x1D6E5}_{1},\unicode[STIX]{x1D6E5}_{2}\in {\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$, we write $\unicode[STIX]{x1D6E5}_{2}\prec \unicode[STIX]{x1D6E5}_{1}$ if $\unicode[STIX]{x1D6E5}_{i}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{i}}^{(l_{i})}$ with $\unicode[STIX]{x1D70C}_{2}|\cdot |^{(1-l_{2})/2+\unicode[STIX]{x1D6FC}}=\unicode[STIX]{x1D70C}_{1}|\cdot |^{(1-l_{1})/2}$ for some $\unicode[STIX]{x1D6FC}\in \mathbb{Z}_{{>}0}$ such that $l_{2}-l_{1}<\unicode[STIX]{x1D6FC}\leqslant l_{2}$. If either $\unicode[STIX]{x1D6E5}_{2}\prec \unicode[STIX]{x1D6E5}_{1}$ or $\unicode[STIX]{x1D6E5}_{1}\prec \unicode[STIX]{x1D6E5}_{2}$, then we say that $\unicode[STIX]{x1D6E5}_{1}$ and $\unicode[STIX]{x1D6E5}_{2}$ are linked. The induced representation $Z(\unicode[STIX]{x1D6E5}_{1})\times Z(\unicode[STIX]{x1D6E5}_{2})$ is reducible if and only if $\unicode[STIX]{x1D6E5}_{1}$ and $\unicode[STIX]{x1D6E5}_{2}$ are linked.

The well-known classification result of Zelevinsky [Reference ZelevinskyZel80, Theorem 6.5] extends the map $\unicode[STIX]{x1D6E5}\mapsto Z(\unicode[STIX]{x1D6E5})$ to a degree-preserving bijection

$$\begin{eqnarray}\mathfrak{m}\mapsto Z(\mathfrak{m})\end{eqnarray}$$

between ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ and $\operatorname{Irr}$. If $\mathfrak{m}=\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k}$ and $\unicode[STIX]{x1D6E5}_{i}\not \prec \unicode[STIX]{x1D6E5}_{j}$ for any $i<j$ (which can always be arranged), then $Z(\mathfrak{m})=\operatorname{soc}(Z(\unicode[STIX]{x1D6E5}_{1})\times \cdots \times Z(\unicode[STIX]{x1D6E5}_{k}))$. An element of ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ is called a multisegment. We have $Z(\mathfrak{m})^{\vee }=Z(\mathfrak{m}^{\vee })$, where we extend $^{\vee }$ from ${\mathcal{S}}{\mathcal{E}}{\mathcal{G}}$ to ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ additively. For any $\mathfrak{m}_{1},\mathfrak{m}_{2}\in {\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$ we have $Z(\mathfrak{m}_{1}+\mathfrak{m}_{2})\,{\leqslant}_{\operatorname{unq}}Z(\mathfrak{m}_{1})\times Z(\mathfrak{m}_{2})$ [Reference Lapid and MínguezLM16, Proposition 3.5]. In particular, if $Z(\mathfrak{m}_{1})\times Z(\mathfrak{m}_{2})$ is irreducible, then it is equal to $Z(\mathfrak{m}_{1}+\mathfrak{m}_{2})$.

We note the following fact.

(1)

By identifying an irreducible supercuspidal representation with a singleton segment we view ${\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$ as a submonoid of ${\mathcal{M}}({\mathcal{S}}{\mathcal{E}}{\mathcal{G}})$. The map $Z$ restricts to a bijection

(2)$$\begin{eqnarray}{\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})\rightarrow \operatorname{Irr}_{\operatorname{gen}}\!.\end{eqnarray}$$

An element of ${\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$ is called a cuspidal datum. We write $\mathfrak{c}(Z(\mathfrak{m}))=\mathfrak{c}(\mathfrak{m})$. The resulting map

$$\begin{eqnarray}\mathfrak{c}:\operatorname{Irr}\rightarrow {\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})\end{eqnarray}$$

is the supercuspidal support (which of course can be defined without reference to the Zelevinsky classification). The restriction of $\mathfrak{c}$ to $\operatorname{Irr}_{\operatorname{gen}}$ is the inverse of (2).

For any segment $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}}^{(l)}$ let $\unicode[STIX]{x1D6E5}^{-}=\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}|\cdot |^{-1/2}}^{(l-1)}$ denote either the segment obtained by removing the endpoint $e(\unicode[STIX]{x1D6E5})$ of $\unicode[STIX]{x1D6E5}$ if $l>1$ or the empty set otherwise.

Now let $\unicode[STIX]{x1D70E}=Z(\mathfrak{m})$, where $\mathfrak{m}=\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k}$. Let

$$\begin{eqnarray}\mathfrak{m}^{-}=\unicode[STIX]{x1D6E5}_{1}^{-}+\cdots +\unicode[STIX]{x1D6E5}_{k}^{-}\quad \text{(disregarding empty sets)}.\end{eqnarray}$$

Define recursively $\mathfrak{m}^{(0)}=\mathfrak{m}$ and $\mathfrak{m}^{(k)}=(\mathfrak{m}^{(k-1)})^{-}$, $k>0$, with $\mathfrak{m}^{(l)}=0$, $l$ minimal. Let $n_{k}=\deg e(\mathfrak{m}^{(k-1)})$, $k=1,\ldots ,l$, so that $n_{1}+\cdots +n_{l}=\deg \unicode[STIX]{x1D70E}$ and let $\unicode[STIX]{x1D714}_{k}=Z(e(\mathfrak{m}^{(k-1)}))\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n_{k}}$. Let $P=P_{\unicode[STIX]{x1D70E}}=M_{\unicode[STIX]{x1D70E}}\ltimes U_{\unicode[STIX]{x1D70E}}=M\ltimes U$ be the standard parabolic subgroup of type $(n_{l},\ldots ,n_{1})$. By [Reference ZelevinskyZel80, §8.3] the Jordan–Hölder sequence of $J_{P}(\unicode[STIX]{x1D70E})$ admits a unique generic irreducible representation $\unicode[STIX]{x1D714}$ of $M$ and, moreover, $\unicode[STIX]{x1D714}\,{\leqslant}_{\operatorname{unq}}J_{P}(\unicode[STIX]{x1D70E})$. Equivalently (by the uniqueness of the Whittaker model), this means that

(3)$$\begin{eqnarray}\operatorname{Hom}_{N_{M}}(J_{P}(\unicode[STIX]{x1D70E}),\unicode[STIX]{x1D713}_{P})=\operatorname{Hom}_{N}(\unicode[STIX]{x1D70E},\unicode[STIX]{x1D713}_{P})=\operatorname{Hom}_{G}(\unicode[STIX]{x1D70E},\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D713}_{P})\text{ is one-dimensional},\end{eqnarray}$$

where $N$ is the maximal nilpotent group of upper unitriangular matrices and $\unicode[STIX]{x1D713}_{P}$ is a character of $N$ which is trivial on $U$ and non-degenerate on $N_{M}=N\cap M$. (This property determines $P$ uniquely up to association.) Moreover, $\unicode[STIX]{x1D714}=\unicode[STIX]{x1D714}_{l}\otimes \cdots \otimes \unicode[STIX]{x1D714}_{1}$ (see, for example, [Reference Mínguez and SécherreMS14, Lemma 9.17]). (For an arbitrary $P$, $\operatorname{Hom}_{N}(\unicode[STIX]{x1D70E},\unicode[STIX]{x1D713}_{P})$ is finite-dimensional.) We will call the image of $\unicode[STIX]{x1D70E}$ in $\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D713}_{P}$ the Zelevinsky model of $\unicode[STIX]{x1D70E}$. In general, $\unicode[STIX]{x1D70E}\not \leqslant \unicode[STIX]{x1D714}_{l}\times \cdots \times \unicode[STIX]{x1D714}_{1}$. For example, if $\unicode[STIX]{x1D70E}=Z(\{\mathbf{1}_{F^{\ast }},|\cdot |\}+\{\mathbf{1}_{F^{\ast }}\})$, then $l=2$, $\unicode[STIX]{x1D714}_{2}=\mathbf{1}_{F^{\ast }}$, $\unicode[STIX]{x1D714}_{1}=Z(\{|\cdot |\}+\{\mathbf{1}_{F^{\ast }}\})$ and $\unicode[STIX]{x1D714}_{2}\times \unicode[STIX]{x1D714}_{1}$ is irreducible (and generic).

2.3 Ladder representations

A multisegment $\mathfrak{m}$ is called a (strict) ladder if it can be written as $\mathfrak{m}=\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k}$ where $\unicode[STIX]{x1D6E5}_{i+1}\prec \unicode[STIX]{x1D6E5}_{i}$ for all $i=1,\ldots ,k-1$. The corresponding irreducible representation $Z(\mathfrak{m})$ is called a ladder representation.

The Jacquet module of ladder representations was described in [Reference Kret and LapidKL12]. The following lemma is an immediate consequence.

Strictly speaking, the results of [Reference Kret and LapidKL12] are stated in terms of the Langlands classification. However, they are also valid in the form above (for the Zelevinsky classification) by either repeating the arguments, or using the Zelevinsky involution.

2.4 $m$-homogeneous representationsFootnote ²

From now on let $m,n\geqslant 1$ be integers and $G=\operatorname{GL}_{mn}$. We say that $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$ is $m$-homogeneous if $\unicode[STIX]{x1D70E}=Z(\unicode[STIX]{x1D6E5}_{1}+\cdots +\unicode[STIX]{x1D6E5}_{k})$ where each $\unicode[STIX]{x1D6E5}_{i}$ is of length $m$. (If $m=1$ this simply means that $\unicode[STIX]{x1D70E}$ is generic.) We denote by $\operatorname{Irr}_{m\operatorname{-hmgns}}G$ the set of irreducible $m$-homogeneous representations of $G$. For any $\unicode[STIX]{x1D70B}=Z(\{\unicode[STIX]{x1D70C}_{1}\}+\cdots +\{\unicode[STIX]{x1D70C}_{k}\})\in \operatorname{Irr}_{\operatorname{gen}}$, define

$$\begin{eqnarray}\operatorname{Sp}(\unicode[STIX]{x1D70B},m)=Z(\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{1}}^{(m)}+\cdots +\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{k}}^{(m)})\in \operatorname{Irr}.\end{eqnarray}$$

The following result is clear.

Suppose that $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ with $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$. Then, in the notation of (2), $P_{\unicode[STIX]{x1D70E}}=P_{m,n}=M\ltimes U$ is the standard parabolic subgroup of $G$ of type $(\overbrace{n,\ldots ,n}^{m})$, consisting of the block upper triangular matrices with blocks of size $n\times n$. Thus, $M\simeq \overbrace{\operatorname{GL}_{n}\times \cdots \times \operatorname{GL}_{n}}^{m}$.

Just as in the case $m=1$, there are simple building blocks for $m$-homogeneous representations.

Proposition 2.7. Let $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)\in \operatorname{Irr}G$ be $m$-homogeneous. Then there exist $\unicode[STIX]{x1D70B}_{1},\ldots ,\unicode[STIX]{x1D70B}_{t}\in \operatorname{Irr}_{\operatorname{gen}}$ such that the following statements hold.

1. (i) $\unicode[STIX]{x1D70E}_{i}:=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{i},m)$ is a ladder representation for all $i$.

2. (ii) $\operatorname{Sp}(\unicode[STIX]{x1D70B},l)=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{1},l)\times \cdots \times \operatorname{Sp}(\unicode[STIX]{x1D70B}_{t},l)$ for all $l=1,\ldots ,m$. In particular, $\unicode[STIX]{x1D70B}=\unicode[STIX]{x1D70B}_{1}\times \cdots \times \unicode[STIX]{x1D70B}_{t}$ and $\unicode[STIX]{x1D70E}=\unicode[STIX]{x1D70E}_{1}\times \cdots \times \unicode[STIX]{x1D70E}_{t}$.

Moreover, let $Q$ be the maximal standard parabolic subgroup of type $((m-1)n,n)$ and denote by $J_{Q}(\unicode[STIX]{x1D70E})_{;\mathfrak{d}}$ the direct summand of $J_{Q}(\unicode[STIX]{x1D70E})$ pertaining to the supercuspidal data $\mathfrak{d}\in {\mathcal{M}}(\operatorname{Irr}_{\operatorname{cusp}})$ in the second ($\operatorname{GL}_{n}$) factor. Then

$$\begin{eqnarray}J_{Q}(\unicode[STIX]{x1D70E})_{;\mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\otimes \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}.\end{eqnarray}$$

Proof. Write $\unicode[STIX]{x1D70B}=Z(\sum _{i\in I}\{\unicode[STIX]{x1D70C}_{i}\})$ with $\unicode[STIX]{x1D70C}_{i}\in \operatorname{Irr}_{\operatorname{cusp}}$ and let $l\geqslant 1$. We say that a subset $J$ of $I$ is an $l$-chain if it can be written, necessarily uniquely, as $J=\{i_{1},\ldots ,i_{r}\}$ where for all $j=1,\ldots ,r-1$ we have $\unicode[STIX]{x1D70C}_{i_{j}}=\unicode[STIX]{x1D70C}_{i_{j+1}}|\cdot |^{\unicode[STIX]{x1D6FC}_{j}}$ with $\unicode[STIX]{x1D6FC}_{j}\in \{1,\ldots ,l\}$. (For example, for a $1$-chain, $\unicode[STIX]{x1D70C}_{i_{r}},\ldots ,\unicode[STIX]{x1D70C}_{i_{1}}$ is a segment.) Clearly, $J$ is an $l$-chain if and only if $Z(\sum _{j\in J}\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{j}}^{(l)})$ is a ladder representation.

We say that two partitions of $I$ are equivalent if one can be obtained from the other by applying a permutation $\unicode[STIX]{x1D70F}$ of $I$ such that $\unicode[STIX]{x1D70C}_{\unicode[STIX]{x1D70F}(i)}=\unicode[STIX]{x1D70C}_{i}$ for all $i$. It is easy to see that for any $l\geqslant 1$ there exists a partition ${\mathcal{P}}^{(l)}(I)$ of $I$ consisting of $l$-chains, such that for any $J,J^{\prime }\in {\mathcal{P}}^{(l)}(I)$ at least one of the following conditions holds:

1. (i) $\{\unicode[STIX]{x1D70C}_{j}:j\in J\}\subset \{\unicode[STIX]{x1D70C}_{j}:j\in J^{\prime }\}$;

2. (ii) $\{\unicode[STIX]{x1D70C}_{j}:j\in J^{\prime }\}\subset \{\unicode[STIX]{x1D70C}_{j}:j\in J\}$;

3. (iii) for every $j\in J$ and $j^{\prime }\in J^{\prime }$ the segments $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{j}}^{(l)}$ and $\unicode[STIX]{x1D6E5}_{\unicode[STIX]{x1D70C}_{j^{\prime }}}^{(l)}$ are unlinked.

Moreover, ${\mathcal{P}}^{(l)}(I)$ is unique up to equivalence. Indeed, ${\mathcal{P}}^{(l)}(I)$ can be defined inductively by taking a maximal $l$-chain $J$ of $I$ (with respect to inclusion) together with the partition ${\mathcal{P}}^{(l)}(I\setminus J)$. It follows from this description that if $l\leqslant m$, then up to equivalence, ${\mathcal{P}}^{(l)}(J)=\{J^{\prime }\in {\mathcal{P}}^{(l)}(I):J^{\prime }\subset J\}$ for any $J\in {\mathcal{P}}^{(m)}(I)$ and, in particular, ${\mathcal{P}}^{(l)}(I)$ is a refinement of ${\mathcal{P}}^{(m)}(I)$.

For any $J\subset I$, let $\unicode[STIX]{x1D70B}_{J}=Z(\sum _{j\in J}\{\unicode[STIX]{x1D70C}_{j}\})\in \operatorname{Irr}_{\operatorname{gen}}$ and $\unicode[STIX]{x1D70E}_{J}=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m)$. Then $\unicode[STIX]{x1D70E}_{J}$ is an ($m$-homogeneous) ladder representation for any $J\in {\mathcal{P}}^{(m)}(I)$. It follows from the defining property of ${\mathcal{P}}^{(m)}(I)$, (1) and Lemma 2.1 that   is irreducible, hence equals $\unicode[STIX]{x1D70E}$. Likewise, for any $l\leqslant m$, we have   Since we may assume that   for all $J\in {\mathcal{P}}^{(m)}(I)$, we infer that

In particular, $\unicode[STIX]{x1D70B}=\times _{J\in {\mathcal{P}}^{(m)}(I)}\unicode[STIX]{x1D70B}_{J}$.

By [Reference Kret and LapidKL12], we have

$$\begin{eqnarray}\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m-1)|\cdot |^{-1/2}\otimes \unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}\,{\leqslant}_{\operatorname{unq}}J_{((m-1)n_{J},n_{J})}(\unicode[STIX]{x1D70E}_{J})\end{eqnarray}$$

for all $J\in {\mathcal{P}}^{(m)}(I)$ where $n_{J}=\deg \unicode[STIX]{x1D70B}_{J}$. Therefore, by the geometric lemma of Bernstein and Zelevinsky [Reference Bernshtein and ZelevinskyBZ77],

$$\begin{eqnarray}\displaystyle \operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\otimes \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2} & = & \displaystyle \times _{J\in {\mathcal{P}}^{(m)}(I)}\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m-1)|\cdot |^{-1/2}\otimes \times _{J\in {\mathcal{P}}^{(m)}(I)}\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}\nonumber\\ \displaystyle & {\leqslant} & \displaystyle J_{Q}(\unicode[STIX]{x1D70E}).\nonumber\end{eqnarray}$$

On the other hand, suppose that $\unicode[STIX]{x1D70F}_{J},\unicode[STIX]{x1D714}_{J}\in \operatorname{Irr}$ with $\unicode[STIX]{x1D70F}_{J}\otimes \unicode[STIX]{x1D714}_{J}\leqslant J_{P}(\unicode[STIX]{x1D70E}_{J})$, $J\in {\mathcal{P}}^{(m)}(I)$ and

(4)$$\begin{eqnarray}\mathop{\sum }_{J\in {\mathcal{P}}^{(m)}(I)}\mathfrak{c}(\unicode[STIX]{x1D714}_{J})=\mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}.\end{eqnarray}$$

We claim that this is possible only if $\unicode[STIX]{x1D70F}_{J}=\operatorname{Sp}(\unicode[STIX]{x1D70B}_{J},m-1)|\cdot |^{-1/2}$ and $\unicode[STIX]{x1D714}_{J}=\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}$ for all $J$. We prove it by induction on $\deg \unicode[STIX]{x1D70B}$ using the geometric lemma. The base of the induction is trivial. For the induction step, it is enough to prove that if $J$ is a maximal $m$-chain, then $\unicode[STIX]{x1D714}_{J}=\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}$. We use Lemma 2.2. By part (ii), if $J$ is a maximal $m$-chain, then $\unicode[STIX]{x1D714}_{J}$ is generic. For otherwise, since $\mathfrak{c}(\unicode[STIX]{x1D714}_{J})\leqslant \mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}$, there would exist $i\in I$ such that $\unicode[STIX]{x1D70C}_{i}\notin \{\unicode[STIX]{x1D70C}_{j}:j\in J\}$ but $\unicode[STIX]{x1D70C}_{i}|\cdot |\in \{\unicode[STIX]{x1D70C}_{j}:j\in J\}$ in contradiction to the maximality of $J$. On the other hand, by part (iv), if $\unicode[STIX]{x1D70C}|\cdot |\not \leqslant \mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}$, then $\unicode[STIX]{x1D70C}$ can occur in $\mathfrak{c}(\unicode[STIX]{x1D714}_{J})$ at most once for any $J\in {\mathcal{P}}^{(m)}(I)$. It follows from (4) that if $\unicode[STIX]{x1D70C}|\cdot |\not \leqslant \mathfrak{c}(\unicode[STIX]{x1D70B})|\cdot |^{(m-1)/2}$, then $\unicode[STIX]{x1D70C}\leqslant \mathfrak{c}(\unicode[STIX]{x1D714}_{J})$ if and only if $\unicode[STIX]{x1D70C}=\unicode[STIX]{x1D70C}_{j}|\cdot |^{(m-1)/2}$ for some $j\in J$. By part (iii), it now follows that if $J$ is a maximal $m$-chain, then $\mathfrak{c}(\unicode[STIX]{x1D714}_{J})=\sum _{j\in J}\{\unicode[STIX]{x1D70C}_{j}|\cdot |^{(m-1)/2}\}$ and hence $\unicode[STIX]{x1D714}_{J}=\unicode[STIX]{x1D70B}_{J}|\cdot |^{(m-1)/2}$ (since $\unicode[STIX]{x1D714}_{J}$ is generic) as required.

This concludes the proof of the proposition. ◻

By Frobenius reciprocity and [Reference Lapid and MínguezLM16, Corollary 4.10], we have the following corollary.

Corollary 2.10. For any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$,

$$\begin{eqnarray}\operatorname{Sp}(\unicode[STIX]{x1D70B},m)=\operatorname{soc}(\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2})\,{\leqslant}_{\operatorname{ unq}}\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)|\cdot |^{-1/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}.\end{eqnarray}$$

By induction on $m$, we get the following result.

Corollary 2.11. For any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$, $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is a subrepresentation of

$$\begin{eqnarray}\unicode[STIX]{x1D6F1}:=\unicode[STIX]{x1D70B}|\cdot |^{(1-m)/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(3-m)/2}\times \cdots \times \unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}.\end{eqnarray}$$

Equivalently (by passing to the contragredient), $\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$ is a quotient of

$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D6F1}}:=\unicode[STIX]{x1D70B}|\cdot |^{(m-1)/2}\times \unicode[STIX]{x1D70B}|\cdot |^{(m-3)/2}\times \cdots \times \unicode[STIX]{x1D70B}|\cdot |^{(1-m)/2}.\end{eqnarray}$$

3.1 Definition of models

Throughout this section, fix $\unicode[STIX]{x1D70B}\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$, let $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)\in \operatorname{Irr}G$ and let $P=P_{\unicode[STIX]{x1D70E}}=P_{m,n}=M\ltimes U$ be the standard parabolic subgroup of $G$ of type $\overbrace{(n,\ldots ,n)}^{m}$. Let $\bar{U}=\,^{t}U$ be the opposite of $U$. Fix a non-trivial character $\unicode[STIX]{x1D713}$ of $F$. Let $\unicode[STIX]{x1D6F9}$ be the function on $G$ given by

$$\begin{eqnarray}\unicode[STIX]{x1D6F9}(g)=\unicode[STIX]{x1D713}\biggl(\mathop{\sum }_{1\leqslant i<nm:n\nmid i}g_{i,i+1}\biggr).\end{eqnarray}$$

We denote the restriction of $\unicode[STIX]{x1D6F9}$ to a subset $A$ of $G$ by $\unicode[STIX]{x1D6F9}_{A}$. Let $N=N_{mn}$ (respectively, $\bar{N}=\,^{t}N$) be the group of upper (respectively, lower) unitriangular matrices in $G$. Then $\unicode[STIX]{x1D6F9}_{N}$ is a (degenerate) character on $N$ that is trivial on $U$ and non-degenerate on $N_{M}:=N\cap M$. Recall that by (3), $\operatorname{Hom}_{G}(\unicode[STIX]{x1D70E},\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D6F9}_{N})$ is one-dimensional.

Denote by ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$ the image of $\unicode[STIX]{x1D70E}$ in $\operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D6F9}_{N}$, that is, the Zelevinsky model of $\unicode[STIX]{x1D70E}$. By Corollaries 2.10 and 2.11, for any $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$, we have

(5a)$$\begin{eqnarray}(|\det |^{(1-n)/2}\otimes |\det |^{(m-1)(n-1)/2})W_{\operatorname{ Ze}}|_{\operatorname{GL}_{(m-1)n}\times \operatorname{GL}_{n}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N\cap (\operatorname{GL}_{(m-1)n}\times \operatorname{GL}_{n})}}(\operatorname{Sp}(\unicode[STIX]{x1D70B},m-1)\otimes \unicode[STIX]{x1D70B}),\end{eqnarray}$$

(5b)$$\begin{eqnarray}\unicode[STIX]{x1D6FF}_{P}^{-1/2}\unicode[STIX]{x1D6FF}^{\prime }W_{\operatorname{ Ze}}|_{M}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{M}}}(\unicode[STIX]{x1D70B}^{\otimes m}),\end{eqnarray}$$

$\unicode[STIX]{x1D70B}^{\otimes m}=\overbrace{\unicode[STIX]{x1D70B}\otimes \cdots \otimes \unicode[STIX]{x1D70B}}^{m}$

$\unicode[STIX]{x1D6FF}_{P}$

$P$

$\unicode[STIX]{x1D6FF}^{\prime }=\unicode[STIX]{x1D6FF}_{P}^{1/2n}$

$M$

$$\begin{eqnarray}\unicode[STIX]{x1D6FF}^{\prime }(\operatorname{diag}(g_{1},\ldots ,g_{m}))=|\det g_{1}|^{(m-1)/2}|\det g_{2}|^{(m-3)/2}\cdots |\det g_{m}|^{(1-m)/2}.\end{eqnarray}$$

The model ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$ is a particular case of more general models considered in [Reference Mœglin and WaldspurgerMW87] (for any reductive group). Let us recall the setup. Let $\mathfrak{g}=\operatorname{Mat}_{nm,nm}$ be the Lie algebra of $G$ over $F$. For any cocharacter $\unicode[STIX]{x1D711}$ of the diagonal torus $T$, let $\mathfrak{g}=\bigoplus _{j\in \mathbb{Z}}\mathfrak{g}_{j}^{\unicode[STIX]{x1D711}}$ be the corresponding grading

$$\begin{eqnarray}\mathfrak{g}_{j}^{\unicode[STIX]{x1D711}}=\{X\in \mathfrak{g}:\operatorname{Ad}(\unicode[STIX]{x1D711}(s))X=s^{j}X\}\end{eqnarray}$$

and let $\mathfrak{g}_{{\geqslant}j}^{\unicode[STIX]{x1D711}}=\bigoplus _{k\geqslant j}\mathfrak{g}_{k}^{\unicode[STIX]{x1D711}}$, $j\in \mathbb{Z}$, be the corresponding filtration. Let $P_{\unicode[STIX]{x1D711}}$ be the semistandard parabolic subgroup such that $\operatorname{Lie}P_{\unicode[STIX]{x1D711}}=\mathfrak{g}_{{\geqslant}0}^{\unicode[STIX]{x1D711}}$. Then $P_{\unicode[STIX]{x1D711}}=M_{\unicode[STIX]{x1D711}}\ltimes U_{\unicode[STIX]{x1D711}}$ where $M_{\unicode[STIX]{x1D711}}$ is the centralizer of $\unicode[STIX]{x1D711}$, $\operatorname{Lie}M_{\unicode[STIX]{x1D711}}=\mathfrak{g}_{0}^{\unicode[STIX]{x1D711}}$ and $\operatorname{Lie}U_{\unicode[STIX]{x1D711}}=\mathfrak{g}_{{>}0}^{\unicode[STIX]{x1D711}}$. Concretely, if $\unicode[STIX]{x1D711}(s)=\operatorname{diag}(s^{\unicode[STIX]{x1D706}_{1}^{\unicode[STIX]{x1D711}}},\ldots ,s^{\unicode[STIX]{x1D706}_{mn}^{\unicode[STIX]{x1D711}}})$ where $(\unicode[STIX]{x1D706}_{1}^{\unicode[STIX]{x1D711}},\ldots ,\unicode[STIX]{x1D706}_{mn}^{\unicode[STIX]{x1D711}})\in \mathbb{Z}^{mn}$, then

$$\begin{eqnarray}\displaystyle P_{\unicode[STIX]{x1D711}} & = & \displaystyle \{g\in G:g_{i,j}=0\text{ if }\unicode[STIX]{x1D706}_{i}<\unicode[STIX]{x1D706}_{j}\},\nonumber\\ \displaystyle M_{\unicode[STIX]{x1D711}} & = & \displaystyle \{g\in G:g_{i,j}=0\text{ if }\unicode[STIX]{x1D706}_{i}\neq \unicode[STIX]{x1D706}_{j}\},\nonumber\\ \displaystyle U_{\unicode[STIX]{x1D711}} & = & \displaystyle \{g\in G:g_{i,j}=\unicode[STIX]{x1D6FF}_{i,j}\text{ if }\unicode[STIX]{x1D706}_{i}\leqslant \unicode[STIX]{x1D706}_{j}\}.\nonumber\end{eqnarray}$$

Consider the nilpotent $nm\times nm$ matrix $J_{m,n}$ consisting of $m$ lower triangular Jordan blocks of size $n\times n$ each. We say that $\unicode[STIX]{x1D711}$ is of type $(m,n)$ if $\operatorname{Ad}(\unicode[STIX]{x1D711}(s))J_{m,n}=s^{-1}J_{m,n}$, or equivalently, if $\unicode[STIX]{x1D706}_{i}^{\unicode[STIX]{x1D711}}-\unicode[STIX]{x1D706}_{i+1}^{\unicode[STIX]{x1D711}}=1$ for all $i$ not dividing $n$. If $\unicode[STIX]{x1D711}$ is of type $(m,n)$, then $\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}$ is a character of $U_{\unicode[STIX]{x1D711}}$. By [Reference Mœglin and WaldspurgerMW87] (in particular, § II.2) we obtain the following theorem.Footnote ³

Theorem 3.1 [Reference Mœglin and WaldspurgerMW87].

Suppose that $\unicode[STIX]{x1D711}$ is of type $(m,n)$. Then the space

$$\begin{eqnarray}\operatorname{Hom}_{U_{\unicode[STIX]{x1D711}}}(\unicode[STIX]{x1D70E},\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}})=\operatorname{Hom}_{G}(\unicode[STIX]{x1D70E},\operatorname{Ind}_{U_{\unicode[STIX]{x1D711}}}^{G}\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}})\end{eqnarray}$$

is one-dimensional.

In the setting of [Reference Mœglin and WaldspurgerMW87] the data pertaining to Theorem 3.1 is the pair $(\unicode[STIX]{x1D711}^{2},J_{m,n})$. (The more general context of [Reference Mœglin and WaldspurgerMW87] applies to cocharacters which are not necessarily even. However, we will not discuss them here.)

We denote by ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}}(\unicode[STIX]{x1D70E})$ the image of $\unicode[STIX]{x1D70E}$ in $\operatorname{Ind}_{U_{\unicode[STIX]{x1D711}}}^{G}\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}$. It consists of functions that are left equivariant (with respect to some character) under the centralizer of $\unicode[STIX]{x1D6F9}_{U_{\unicode[STIX]{x1D711}}}$ in $P_{\unicode[STIX]{x1D711}}$.

Clearly, any $\unicode[STIX]{x1D711}$ of type $(m,n)$ is determined by the $m$-tuple $(\unicode[STIX]{x1D706}_{n}^{\unicode[STIX]{x1D711}},\unicode[STIX]{x1D706}_{2n}^{\unicode[STIX]{x1D711}},\ldots ,\unicode[STIX]{x1D706}_{mn}^{\unicode[STIX]{x1D711}})$. We consider $(m-1)n+1$ cocharacters $\unicode[STIX]{x1D711}_{0},\ldots ,\unicode[STIX]{x1D711}_{(m-1)n}$ of $T$ of type $(m,n)$ such that $\unicode[STIX]{x1D706}_{nk}^{\unicode[STIX]{x1D711}_{i}}-\unicode[STIX]{x1D706}_{n(k+1)}^{\unicode[STIX]{x1D711}_{i}}=\max (0,nk-i)$, $k=1,\ldots ,m-1$. (Up to a cocharacter of the center of $G$, the cocharacter $\unicode[STIX]{x1D711}_{(m-1)n}^{2}$ corresponds to the $\operatorname{SL}_{2}$-triple pertaining to $J_{m,n}$.) For simplicity we write $P_{i}=P_{\unicode[STIX]{x1D711}_{i}}$, $M_{i}=M_{\unicode[STIX]{x1D711}_{i}}$, $U_{i}=U_{\unicode[STIX]{x1D711}_{i}}$. If $i=nd+r$ where $d=\lfloor i/n\rfloor$ and $0\leqslant r<n$, then $U_{i}$ consists of the matrices whose $n\times n$ blocks $A_{j,k}$ satisfy:

1. (i) $A_{j,j}$ is upper unitriangular for all $j=1,\ldots ,m$;

2. (ii) $A_{j,k}$ is strictly upper triangular if $j\neq k$ and $j,k\leqslant d+1$;

3. (iii) for any $k<d+2$, $(A_{d+2,k})_{a,b}=0$ if $b-a\leqslant n-r$ and $(A_{k,d+2})_{a,b}=0$ if $a-b\geqslant n-r$;

4. (iv) $A_{j,k}=0$ if $j>k$ and $j>d+2$.

(There is no constraint on $A_{j,k}$ if $j<k$ and $d+2<k$.)

In particular, $U_{0}=N$ while $U_{(m-1)n}$ consists of the matrices whose difference from the identity matrix is strictly upper triangular in each $n\times n$ block. Also, $U_{i+1}\cap N\subset U_{i}\cap N$ and $U_{i}\cap \bar{N}\subset U_{i+1}\cap \bar{N}$ for all $i$.

For brevity we write $P^{\prime }=P_{(m-1)n}$, $M^{\prime }=M_{(m-1)n}\simeq \overbrace{\operatorname{GL}_{m}\times \cdots \times \operatorname{GL}_{m}}^{n}$, $U^{\prime }=U_{(m-1)n}$. In analogy with the case $m=2$ we will refer to ${\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$ as the Shalika model of $\unicode[STIX]{x1D70E}$. We caution, however, that in the literature, this terminology usually refers to the image of $\unicode[STIX]{x1D70F}\in \operatorname{Irr}\operatorname{GL}_{2n}$ (possibly generic) in $\operatorname{Ind}_{S}^{\operatorname{GL}_{2n}}\unicode[STIX]{x1D713}_{S}$ under a non-trivial intertwining operator, if it exists (in which case it is unique up to a scalar [Reference Jacquet and RallisJR96]), where $S$ is the Shalika group

$$\begin{eqnarray}S=\big\{\!\left(\begin{smallmatrix}g & \\ & g\end{smallmatrix}\right)\left(\begin{smallmatrix}I_{n} & X\\ & I_{n}\end{smallmatrix}\right):g\in \operatorname{GL}_{n},X\in \operatorname{Mat}_{n,n}\!\big\}\end{eqnarray}$$

and $\unicode[STIX]{x1D713}_{S}$ is the character on $S$ given by $\unicode[STIX]{x1D713}(\operatorname{tr}X)$. In the case at hand, any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$ automatically satisfies an equivariance property under the centralizer of $\unicode[STIX]{x1D6F9}_{U^{\prime }}$ in $P^{\prime }$ (which is conjugate to $S$ in the case $m=2$), which justifies our terminology. In general, even for $m=2$, $\operatorname{Hom}_{U^{\prime }}(\unicode[STIX]{x1D70F},\unicode[STIX]{x1D6F9}_{U^{\prime }})$ is infinite-dimensional for $\unicode[STIX]{x1D70F}\in \operatorname{Irr}G$.

Letting $G$ act on right on the vector space $F^{mn}$ of row vectors with standard basis $e_{1},\ldots ,e_{mn}$, $P^{\prime }$ is the stabilizer of the flag

$$\begin{eqnarray}(\operatorname{span}\{e_{nj-k}:j=1,\ldots ,m,~k=0,\ldots ,i-1\})_{i=0,\ldots ,n}.\end{eqnarray}$$

We denote by $\unicode[STIX]{x1D705}:\overbrace{\operatorname{GL}_{m}\times \cdots \times \operatorname{GL}_{m}}^{n}\rightarrow M^{\prime }$ the isomorphism such that the $i$th copy of $\operatorname{GL}_{m}$ acts on $\operatorname{span}\{e_{nj+i}:j=0,\ldots ,m-1\}$.

If $X$ is a matrix over $F$, then we write $\Vert X\Vert$ for the maximum of the absolute value of its entries.

3.2 Model transition part I

We denote by $M_{i}^{\unicode[STIX]{x1D6F9}}$ (respectively, $P_{i}^{\unicode[STIX]{x1D6F9}}=M_{i}^{\unicode[STIX]{x1D6F9}}\ltimes U_{i}$) the stabilizer of $\unicode[STIX]{x1D6F9}_{U_{i}}$ in $M_{i}$ (respectively, $P_{i}$). (Note that $M_{i}$ determines $i$ so this notation is unambiguous.) We also write ${M^{\prime }}^{\unicode[STIX]{x1D6F9}}=M_{(m-1)n}^{\unicode[STIX]{x1D6F9}}$ and ${P^{\prime }}^{\unicode[STIX]{x1D6F9}}=P_{(m-1)n}^{\unicode[STIX]{x1D6F9}}={M^{\prime }}^{\unicode[STIX]{x1D6F9}}\ltimes U^{\prime }$. Note that ${P^{\prime }}^{\unicode[STIX]{x1D6F9}}$ is unimodular. Explicitly, ${M^{\prime }}^{\unicode[STIX]{x1D6F9}}$ is the image under $\unicode[STIX]{x1D705}$ of $\operatorname{GL}_{m}$ diagonally embedded in $\overbrace{\operatorname{GL}_{m}\times \cdots \times \operatorname{GL}_{m}}^{n}$. It consists of the matrices in $G$ whose $n\times n$ blocks are all scalar matrices. Let $\unicode[STIX]{x1D704}:\operatorname{GL}_{m}\rightarrow {M^{\prime }}^{\unicode[STIX]{x1D6F9}}$ be the resulting identification.

In general, write $i=nd+r$, $0\leqslant r<n$. Then the reductive part of $M_{i}^{\unicode[STIX]{x1D6F9}}$ is the image under $\unicode[STIX]{x1D704}$ of the subgroup

$$\begin{eqnarray}\{\operatorname{diag}(l,t_{d+2},\ldots ,t_{m}):l\in \operatorname{GL}_{d+1},t_{d+2},\ldots ,t_{m}\in F^{\ast }\}.\end{eqnarray}$$

The unipotent radical of $M_{i}^{\unicode[STIX]{x1D6F9}}$ consists of the matrices whose $n\times n$ blocks $A_{j,k}$ satisfy the following requirements.

1. (i) $A_{j,j}=I_{n}$ for all $j$.

2. (ii) If $j\neq k$, then $A_{j,k}=0$ unless $k=d+2$ and $j<k$, in which case $(A_{j,k})_{a,b}=0$ unless $a-b=n-r$. Moreover, all the entries of $A_{j,k}$ on the diagonal $a-b=n-r$ coincide.

This group is trivial if $i$ is divisible by $n$, and is of dimension $d+1$ otherwise.

Proof. For any $j,k$, we have $\unicode[STIX]{x1D706}_{j}^{\unicode[STIX]{x1D711}_{i}}-\unicode[STIX]{x1D706}_{k}^{\unicode[STIX]{x1D711}_{i}}-(\unicode[STIX]{x1D706}_{j}^{\unicode[STIX]{x1D711}_{i+1}}-\unicode[STIX]{x1D706}_{k}^{\unicode[STIX]{x1D711}_{i+1}})\in \{-1,0,1\}$. It follows that

(7)$$\begin{eqnarray}\mathfrak{g}_{{\geqslant}j+1}^{\unicode[STIX]{x1D711}_{i+1}}\subset \mathfrak{g}_{{\geqslant}j}^{\unicode[STIX]{x1D711}_{i}}\subset \mathfrak{g}_{{\geqslant}j-1}^{\unicode[STIX]{x1D711}_{i+1}}\quad \text{for all }j.\end{eqnarray}$$

Therefore, $U_{i}\subset P_{i+1}$ and $U_{i+1}\subset P_{i}$. Hence, $U_{i}$ and $U_{i+1}$ normalize each other, so that $U_{i}\cdot U_{i+1}$ is a subgroup of $G$ that contains $U_{i}$ and $U_{i+1}$ as normal subgroups. The equalities (6) are now clear since $T\subset M_{i},M_{i+1}$. By (7), we have $\operatorname{Lie}M_{i+1}\cap U_{i}=\mathfrak{g}_{{>}0}^{\unicode[STIX]{x1D711}_{i}}\cap \mathfrak{g}_{0}^{\unicode[STIX]{x1D711}_{i+1}}\subset \mathfrak{g}_{1}^{\unicode[STIX]{x1D711}_{i}}$. It follows that $M_{i+1}\cap U_{i}$ is abelian since $[\mathfrak{g}_{1}^{\unicode[STIX]{x1D711}_{i}},\mathfrak{g}_{1}^{\unicode[STIX]{x1D711}_{i}}]\subset \mathfrak{g}_{2}^{\unicode[STIX]{x1D711}_{i}}$. Similarly, $M_{i}\cap U_{i+1}$ is abelian. The rest of the lemma follows easily from the fact that $U_{i+1}\cap M_{i}^{\unicode[STIX]{x1D6F9}}=1$.◻

In the rest of the section we endow various unipotent subgroups of $G$ with Haar measures. Thanks to the choice of basis $e_{1},\ldots ,e_{mn}$, the Lie algebra of any of these unipotent groups has a natural basis as a vector space over $F$. Our convention will be to take the measure corresponding to the product measure where the Haar measure on $F$ is the one which is self-dual with respect to $\unicode[STIX]{x1D713}$.

The following is a special case of [Reference Gomez, Gourevitch and SahiGGS16] (see also [Reference Gomez, Gourevitch and SahiGGS17]). For future reference and in order to be self-contained we provide the (elementary) proof. We refer the reader to [Reference Gomez, Gourevitch and SahiGGS16, Reference Gomez, Gourevitch and SahiGGS17] for a more thorough discussion about interplay between models.

Proposition 3.5. For any $i=0,\ldots ,(m-1)n-1$, the map

(8)$$\begin{eqnarray}\displaystyle W_{i}\mapsto \int _{U_{i}\cap U_{i+1}\backslash U_{i+1}}W_{i}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U_{i+1}}(u^{\prime })^{-1}\,du^{\prime } & = & \displaystyle \int _{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}W_{i}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U_{i+1}}(u^{\prime })^{-1}\,du^{\prime }\nonumber\\ \displaystyle & = & \displaystyle \int _{U_{i}\cap \bar{N}\backslash U_{i+1}\cap \bar{N}}W_{i}(u^{\prime }\cdot )\,du^{\prime }\end{eqnarray}$$

defines an isomorphism ${\mathcal{T}}_{i}={\mathcal{T}}_{i}^{\unicode[STIX]{x1D713}}:{\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})\rightarrow {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i+1}}}(\unicode[STIX]{x1D70E})$. Its inverse is given by

(9)$$\begin{eqnarray}W_{i+1}\mapsto \int _{U_{i}\cap P_{i+1}^{\unicode[STIX]{x1D6F9}}\backslash U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du.\end{eqnarray}$$

In both cases the integrands are compactly supported.

Proof. For any $W_{i}\in \operatorname{Ind}_{U_{i}}^{G}\unicode[STIX]{x1D6F9}_{U_{i}}$, $u\in U_{i}$ and $u^{\prime }\in U_{i+1}$, we have $W_{i}(u^{\prime }u)=\mathfrak{c}_{i}(u)(u^{\prime })\unicode[STIX]{x1D6F9}_{U_{i}}(u)W_{i}(u^{\prime })$. It follows from Lemma 3.3 and the smoothness of $W_{i}$ that $W_{i}|_{U_{i+1}}$ is compactly supported modulo $U_{i}\cap U_{i+1}$ and that, for any $u\in U_{i}$,

(10)

Recall that any $W_{i+1}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i+1}}}(\unicode[STIX]{x1D70E})$ is left-$M_{i+1}^{\unicode[STIX]{x1D6F9}}$-equivariant under a character and, in particular, is left-invariant under any unipotent subgroup of $M_{i+1}^{\unicode[STIX]{x1D6F9}}$. Also, $U_{i}\cap P_{i+1}^{\unicode[STIX]{x1D6F9}}=(U_{i}\cap M_{i+1}^{\unicode[STIX]{x1D6F9}})\ltimes (U_{i}\cap U_{i+1})$. By similar reasoning as before, $W_{i+1}|_{U_{i}}$ is compactly supported modulo $U_{i}\cap P_{i+1}^{\unicode[STIX]{x1D6F9}}$. By Lemma 3.3 and Fourier inversion, the map (9) defines a $G$-equivariant left inverse to ${\mathcal{T}}_{i}$. Since the spaces are irreducible, it is also a right inverse.◻

We write

$$\begin{eqnarray}{\mathcal{T}}={\mathcal{T}}^{\unicode[STIX]{x1D713}}={\mathcal{T}}_{(m-1)n-1}\circ \ldots \circ {\mathcal{T}}_{0}:{\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})\rightarrow {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E}).\end{eqnarray}$$

This operator was considered in [Reference Cai, Friedberg and KaplanCFK18, §2.4].

3.3 Model transition part II

We now introduce a subgroup of $G$ that will play an important role in what follows. Let $D=D_{m,n}$ be the joint stabilizer of the vectors $e_{jn}$, $j=1,\ldots ,m$, in $G$ and let $N_{D}=N\cap D\supset N_{M}$. Note that $U^{\prime }\subset D$. (In the case $m=1$, $D$ is the standard mirabolic subgroup.)

The following lemma is straightforward.

Hence, we can rewrite (9) as

$$\begin{eqnarray}W_{i+1}\mapsto \int _{D\cap U_{i}\cap U_{i+1}\backslash D\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du=\int _{N_{D}\cap U_{i+1}\backslash N_{D}\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du.\end{eqnarray}$$

Lemma 3.8. Any $W_{\operatorname{Ze}}\in \operatorname{Ind}_{N}^{G}\unicode[STIX]{x1D6F9}_{N}$ is compactly supported on $D\cap \bar{N}$. Hence,

$$\begin{eqnarray}{\mathcal{T}}W_{\operatorname{Ze}}=\int _{U^{\prime }\cap N\backslash U^{\prime }}W_{\operatorname{Ze}}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U^{\prime }}(u^{\prime })^{-1}\,du^{\prime }=\int _{U^{\prime }\cap \bar{N}}W_{\operatorname{Ze}}(u^{\prime }\cdot )\,du^{\prime }=\int _{U^{\prime }\cap \bar{U}}W_{\operatorname{Ze}}(u^{\prime }\cdot )\,du^{\prime }\end{eqnarray}$$

where the integrand is compactly supported.

Recall that any $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$ is $M_{i}^{\unicode[STIX]{x1D6F9}}$-equivariant with respect to some character $\unicode[STIX]{x1D712}_{i}$ of $M_{i}^{\unicode[STIX]{x1D6F9}}$ (depending only on $\unicode[STIX]{x1D70E}$). As in [Reference Cai, Friedberg, Ginzburg and KaplanCFGK19, Reference Cai, Friedberg and KaplanCFK18] we can easily explicate this character.

Lemma 3.9 (Cf. [Reference Cai, Friedberg, Ginzburg and KaplanCFGK19, Proposition 24], [Reference Cai, Friedberg and KaplanCFK18, §2.6]).

For any $i=nd+r$, $0\leqslant r<n$, $l\in \operatorname{GL}_{d+1}$ and $t_{d+2},\ldots ,t_{m}\in F^{\ast }$, we have

$$\begin{eqnarray}\displaystyle & & \displaystyle \unicode[STIX]{x1D712}_{i}(\unicode[STIX]{x1D704}(\operatorname{diag}(l,t_{d+2},\ldots ,t_{m})))\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}(t_{d+2}\ldots t_{m}\det l)|\det l|^{-\binom{r}{2}+\binom{n}{2}(m-d-1)}|t_{d+2}|^{\binom{r}{2}(d+1)}\mathop{\prod }_{j=d+2}^{m}|t_{j}|^{n(n-1)((m+1)/2-j)}\nonumber\end{eqnarray}$$

where $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}$ is the central character of $\unicode[STIX]{x1D70B}$. In particular, for any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$,

$$\begin{eqnarray}W_{\operatorname{Sh}}(\unicode[STIX]{x1D704}(l)g)=\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}(\det l)W_{\operatorname{Sh}}(g)\quad \forall l\in \operatorname{GL}_{m},~g\in G.\end{eqnarray}$$

Proof. It is enough to evaluate $\unicode[STIX]{x1D712}_{i}$ on an element $\unicode[STIX]{x1D704}(t)$ where $t=\operatorname{diag}(t_{1},\ldots ,t_{m})$ is in the diagonal torus of $\operatorname{GL}_{m}$. Note that $\unicode[STIX]{x1D704}(t)$ lies in the center $Z_{M}$ of $M$. Writing $W_{i}=\int _{\bar{N}\cap U_{i}}W_{\operatorname{Ze}}(u\cdot )\,du$ with $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$ (the integrand is compactly supported by Lemma 3.8), the required relation follows from the equality

$$\begin{eqnarray}\displaystyle & & \displaystyle \unicode[STIX]{x1D6FF}_{P}^{1/2}{\unicode[STIX]{x1D6FF}^{\prime }}^{-1}(\unicode[STIX]{x1D704}(t))=\mathop{\prod }_{j=1}^{m}|t_{j}|^{n(n-1)((m+1)/2-j)}\nonumber\\ \displaystyle & & \displaystyle \quad =\bigg(\mathop{\prod }_{j=1}^{d+1}|t_{j}|^{-\binom{r}{2}+\binom{n}{2}(m-d-1)}\bigg)\bigg(\mathop{\prod }_{j=d+2}^{m}|t_{j}|^{n(n-1)((m+1)/2-j)}\bigg)|t_{d+2}|^{\binom{r}{2}(d+1)}\unicode[STIX]{x1D6FF}_{Z_{M}\ltimes (U_{i}\cap \bar{N})}(\unicode[STIX]{x1D704}(t))^{-1}.\nonumber\end{eqnarray}$$

◻

Remark 3.10. Let   (alternating signs on the non-principal diagonal) and $\tilde{w}_{m,n}=\unicode[STIX]{x1D704}(\tilde{w}_{m})$. By Lemma 3.9, we have

$$\begin{eqnarray}W_{\operatorname{Sh}}(\tilde{w}_{m,n}g)=W_{\operatorname{Sh}}(g)\end{eqnarray}$$

for any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$.

Lemma 3.11. The inverse of ${\mathcal{T}}$ is given by

(11)$$\begin{eqnarray}W_{\operatorname{Sh}}\mapsto \int _{N\cap U^{\prime }\backslash N_{D}}W_{\operatorname{Sh}}(u\cdot )\unicode[STIX]{x1D6F9}_{N}(u)^{-1}\,du\end{eqnarray}$$

where the integrand is compactly supported.

Proof. From Proposition 3.5 we only need to check that the integrand is compactly supported. Assume that $W_{\operatorname{Sh}}={\mathcal{T}}W_{\operatorname{Ze}}$. By Remark 3.10, the integral equals

$$\begin{eqnarray}\int _{N\cap U^{\prime }\backslash N_{D}}W_{\operatorname{Sh}}(\tilde{w}_{m,n}u\cdot )\unicode[STIX]{x1D6F9}_{N}(u)^{-1}\,du=\int _{U\cap U^{\prime }\backslash U_{D}}\biggl(\int _{U^{\prime }\cap \bar{U}}W_{\operatorname{Ze}}(v\tilde{w}_{m,n}u\cdot )\unicode[STIX]{x1D6F9}_{U^{\prime }}(v)^{-1}\unicode[STIX]{x1D6F9}_{N}(u)^{-1}\,dv\biggr)du\end{eqnarray}$$

where $U_{D}=U\cap D$. The latter double integral is

$$\begin{eqnarray}\int _{\bar{U}\cap U^{\prime }\backslash \bar{U}_{D}}\biggl(\int _{U^{\prime }\cap \bar{U}}W_{\operatorname{Ze}}(v\bar{u}\tilde{w}_{m,n}\cdot )\unicode[STIX]{x1D6F9}_{U^{\prime }}(v)^{-1}\unicode[STIX]{x1D6F9}_{N}({\tilde{w}_{m,n}}^{-1}\bar{u}\tilde{w}_{m,n})^{-1}\,dv\biggr)\,d\bar{u}.\end{eqnarray}$$

By Lemma 3.8 the integrand is compactly supported in $v$, $\bar{u}$. Thus, the integrand on the right-hand side of (11) is compactly supported.◻

3.4 Kirillov–Shalika model

The following lemma is an analogue of [Reference Gel’fand and KajdanGK75, Proposition 2].

The proof of [Reference Gel’fand and KajdanGK75, pp. 110–111] (for the case $m=1$) applies verbatim. One only needs to observe that the unipotent radical $V$ of $D$ is abelian, the stabilizer of the character $\unicode[STIX]{x1D6F9}_{V}$ under the action of $D$ modulo $V$ is isomorphic to $D_{m,n-1}$ and the map $p\mapsto \unicode[STIX]{x1D6F9}_{V}(p^{-1}\cdot p)$ defines an open map from $D$ to the Pontryagin dual of $V$.

Let $Q$ be the stabilizer of $\operatorname{span}\{e_{ni}:i=1,\ldots ,m\}$ in $G$ – a maximal parabolic subgroup of $G$ of type $((n-1)m,m)$. Thus, $Q=D\rtimes {M^{\prime }}^{\unicode[STIX]{x1D6F9}}$ and $\unicode[STIX]{x1D6FF}_{Q}|_{{M^{\prime }}^{\unicode[STIX]{x1D6F9}}}\equiv 1$.

Corollary 3.13. For any $m$-homogeneous $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$, the image of the restriction map

(12)$$\begin{eqnarray}W_{\operatorname{Sh}}\mapsto W_{\operatorname{Sh}}|_{D},\quad {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})\rightarrow \operatorname{Ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}\end{eqnarray}$$

contains $\operatorname{ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$. Equivalently, by Lemmas 2.3 and 3.9, if $\unicode[STIX]{x1D70E}=\operatorname{Sp}(\unicode[STIX]{x1D70B},m)$, then the image ${\mathcal{K}}^{\unicode[STIX]{x1D713}}(\unicode[STIX]{x1D70E})$ of the restriction map

$$\begin{eqnarray}W_{\operatorname{Sh}}\mapsto W_{\operatorname{Sh}}|_{Q},\quad {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})\rightarrow \operatorname{Ind}_{{P^{\prime }}^{\unicode[STIX]{x1D6F9}}}^{Q}\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}\end{eqnarray}$$

contains $\operatorname{ind}_{{P^{\prime }}^{\unicode[STIX]{x1D6F9}}}^{Q}\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}$ where $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}$ is the character of ${P^{\prime }}^{\unicode[STIX]{x1D6F9}}$ such that $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}|_{U^{\prime }}=\unicode[STIX]{x1D6F9}_{U^{\prime }}$ and $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}^{\unicode[STIX]{x1D6F9}}\circ \unicode[STIX]{x1D704}=\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}\circ \det$.

We will call ${\mathcal{K}}^{\unicode[STIX]{x1D713}}(\unicode[STIX]{x1D70E})$ the Kirillov–Shalika model of $\unicode[STIX]{x1D70E}$.

Lemma 3.14. For any $i=0,\ldots ,(m-1)n-1$, the map

$$\begin{eqnarray}\tilde{{\mathcal{T}}}_{i}:W_{i}\mapsto \int _{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}W_{i}(u^{\prime }\cdot )\unicode[STIX]{x1D6F9}_{U_{i+1}}(u^{\prime })^{-1}\,du^{\prime }=\int _{U_{i}\cap \bar{N}\backslash U_{i+1}\cap \bar{N}}W_{i}(u^{\prime }\cdot )\,du^{\prime }\end{eqnarray}$$

is an isomorphism between $\operatorname{Ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$ and $\operatorname{Ind}_{D\cap U_{i+1}}^{D}\unicode[STIX]{x1D6F9}_{U_{i+1}}$, whose inverse is given by

$$\begin{eqnarray}W_{i+1}\mapsto \int _{D\cap U_{i}\cap U_{i+1}\backslash D\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du=\int _{N_{D}\cap U_{i+1}\backslash N_{D}\cap U_{i}}W_{i+1}(u\cdot )\unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}\,du.\end{eqnarray}$$

Moreover,

$$\begin{eqnarray}\tilde{{\mathcal{T}}}_{i}(\operatorname{ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}})=\operatorname{ind}_{D\cap U_{i+1}}^{D}\unicode[STIX]{x1D6F9}_{U_{i+1}}.\end{eqnarray}$$

Finally, the $D$-module $\operatorname{ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$ is irreducible.

Proof. Let $W_{i}\in \operatorname{Ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$. As in the proof of Proposition 3.5, by Lemma 3.7 the function

$$\begin{eqnarray}u\in D\cap U_{i}\cap U_{i+1}\backslash D\cap U_{i}\mapsto \unicode[STIX]{x1D6F9}_{U_{i}}(u)^{-1}{\mathcal{T}}_{i}W_{i}(u)\end{eqnarray}$$

is the Fourier transform of the function $W_{i}\unicode[STIX]{x1D6F9}_{U_{i+1}}^{-1}|_{U_{i}\cap U^{\prime }\backslash U_{i+1}\cap U^{\prime }}$ at $\mathfrak{c}_{i}(u)$. The first claim follows by Fourier inversion.

Suppose that $W_{i}\in \operatorname{ind}_{D\cap U_{i}}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$. From the definition (and since $U^{\prime }\subset D$), $\tilde{{\mathcal{T}}}_{i}W_{i}$ is supported on $(D\cap U_{i}\cdot U_{i+1})\unicode[STIX]{x1D6FA}$, where $\unicode[STIX]{x1D6FA}$ is a compact subset of $D$. Fix $g\in \unicode[STIX]{x1D6FA}$. It follows from the above that the function $\tilde{{\mathcal{T}}}_{i}W_{i}(g)$ is compactly supported modulo $D\cap U_{i}\cap U_{i+1}$. Hence, $\tilde{{\mathcal{T}}}_{i}W_{i}$ is compactly supported modulo $D\cap U_{i+1}$.

The last part now follows from the fact that $\operatorname{ind}_{U^{\prime }}^{D}\unicode[STIX]{x1D6F9}_{U^{\prime }}$ is irreducible.◻

From Lemma 3.14, Proposition 3.5 and Corollary 3.13 we obtain the following result.

Corollary 3.15. For any $m$-homogeneous $\unicode[STIX]{x1D70E}\in \operatorname{Irr}G$ and for any $i=0,\ldots ,(m-1)n$, the image of the restriction map

(13)$$\begin{eqnarray}W_{i}\mapsto W_{i}|_{D},\quad {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})\rightarrow \operatorname{Ind}_{U_{i}\cap D}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}\end{eqnarray}$$

contains $\operatorname{ind}_{U_{i}\cap D}^{D}\unicode[STIX]{x1D6F9}_{U_{i}}$.

Once again, in analogy with the case $m=1$ (conjectured in [Reference Gel’fand and KajdanGK75], proved in [Reference Bernšten̆ and ZelevinskyBZ76, Reference Bernshtein and ZelevinskyBZ77]) it is natural to make the following conjecture.

We will prove a special case in Corollary 4.4 below.

We do not know whether, in general, the restriction of $\unicode[STIX]{x1D70E}$ to $Q$ is of finite length. (See Proposition 7.1 for a very special case.) Recall that in the case $m=1$ this is known (for any $\unicode[STIX]{x1D70B}\in \operatorname{Irr}$, not necessarily generic) using the theory of derivatives of Bernstein and Zelevinsky [Reference Bernšten̆ and ZelevinskyBZ76, Reference Bernshtein and ZelevinskyBZ77]. It would be very interesting to have an analogous theory for $m>1$.

5.1 Statement of the result

We write an analogue of the Rankin–Selberg integral for $\unicode[STIX]{x1D70E}\times \unicode[STIX]{x1D70E}^{\prime }$ on the Shalika model as follows. Recall that $\unicode[STIX]{x1D702}\in \operatorname{Mat}_{m,nm}(F)$ is the matrix whose $i$th row is $e_{ni}$, $i=1,\ldots ,m$, so that $D$ is the stabilizer of $\unicode[STIX]{x1D702}$ in $G$. For any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$, $W_{\operatorname{Sh}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$, $\unicode[STIX]{x1D6F7}\in {\mathcal{S}}(\operatorname{Mat}_{m,nm}(F))$, consider

$$\begin{eqnarray}Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)=\int _{U^{\prime }\backslash G}W_{\operatorname{Sh}}(g)W_{\operatorname{Sh}}^{\prime }(g)\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{s}\,dg.\end{eqnarray}$$

This expression was already considered in some form in the proof of Proposition 4.1.

Note that in the case $n=1$ (where $U^{\prime }=1$) $Z$ reduces to the generalized Tate integral for (a character of) $\operatorname{GL}_{m}$ considered by Godement and Jacquet [Reference Godement and JacquetGJ72].

For any $k$, let

We will prove the theorem below by relating $Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)$ to the usual Rankin–Selberg integrals.

5.2 A result of Jacquet, Piatetski-Shapiro and Shalika

Recall the $\operatorname{GL}_{n}\times \operatorname{GL}_{n}$ local Rankin–Selberg integrals studied by Jacquet, Piatetski-Shapiro and Shalika [Reference Jacquet, Piatetskii-Shapiro and ShalikaJPSS83]. They are given by

$$\begin{eqnarray}Z^{\operatorname{GL}_{n}}(W,W^{\prime },\unicode[STIX]{x1D6F7},s)=\int _{N_{n}\backslash \!\operatorname{GL}_{n}}W(g)W^{\prime }(g)\unicode[STIX]{x1D6F7}(e_{n}g)|\det g|^{s}\,dg\end{eqnarray}$$

where $W\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{n}}}(\unicode[STIX]{x1D70B})$, $W^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{n}}^{-1}}(\unicode[STIX]{x1D70B}^{\prime })$, $\unicode[STIX]{x1D6F7}\in {\mathcal{S}}(F^{n})$ and $s\in \mathbb{C}$. The integral converges for $\operatorname{Re}s+\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\prime })>0$ and admits a meromorphic continuation in $s$ to a rational function in $q^{s}$. The quotient

$$\begin{eqnarray}\frac{Z^{\operatorname{GL}_{n}}(W,W^{\prime },\unicode[STIX]{x1D6F7},s)}{L(s,\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70B}^{\prime })}\end{eqnarray}$$

is a Laurent polynomial in $q^{s}$ which can be made non-zero at any given $s\in \mathbb{C}$ by an appropriate choice of $W$, $W^{\prime }$, $\unicode[STIX]{x1D6F7}$. Moreover, we have a functional equation

$$\begin{eqnarray}Z^{\operatorname{GL}_{n}}(\widehat{W},\widehat{W^{\prime }},\hat{\unicode[STIX]{x1D6F7}},1-s)=\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}^{\prime }}(-1)^{n-1}\unicode[STIX]{x1D6FE}(s,\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70B}^{\prime },\unicode[STIX]{x1D713})Z^{\operatorname{GL}_{n}}(W,W^{\prime },\unicode[STIX]{x1D6F7},s)\end{eqnarray}$$

where $\widehat{W}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{n}}}(\unicode[STIX]{x1D70B}^{\vee })$, $\widehat{W^{\prime }}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{n}}^{-1}}({\unicode[STIX]{x1D70B}^{\prime }}^{\vee })$ are given by

$$\begin{eqnarray}\widehat{W}(g)=W(w_{n}\,^{t}g^{-1}),\quad \widehat{W^{\prime }}(g)=W^{\prime }(w_{n}\,^{t}g^{-1})\end{eqnarray}$$

and $\hat{\unicode[STIX]{x1D6F7}}$ is the Fourier transform of $\unicode[STIX]{x1D6F7}$ given by

$$\begin{eqnarray}\hat{\unicode[STIX]{x1D6F7}}(y)=\int _{F^{n}}\unicode[STIX]{x1D6F7}(x)\unicode[STIX]{x1D713}(\left\langle x,y\right\rangle )\,dx\end{eqnarray}$$

where $\left\langle (x_{1},\ldots ,x_{n}),(y_{1},\ldots ,y_{n})\right\rangle =\sum _{i}x_{i}y_{i}$ denotes the standard pairing on $F^{n}$.

Slightly more generally, for $W\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{M}}}(\unicode[STIX]{x1D70B}^{\otimes m})$, $W^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{M}}^{-1}}(\unicode[STIX]{x1D70B}^{\prime \otimes m})$, $\tilde{\unicode[STIX]{x1D6F7}}\in {\mathcal{S}}((F^{n})^{m})$ and $(s_{1},\ldots ,s_{m})\in \mathbb{C}^{m}$, we write

$$\begin{eqnarray}Z^{M}(W,W^{\prime },\tilde{\unicode[STIX]{x1D6F7}},(s_{1},\ldots ,s_{m}))=\int _{N_{M}\backslash M}W(l)W^{\prime }(l)\tilde{\unicode[STIX]{x1D6F7}}(e_{n}g_{1},\ldots ,e_{n}g_{n})\mathop{\prod }_{i=1}^{m}|\det g_{i}|^{s_{i}}\,dl\end{eqnarray}$$

where $l=\operatorname{diag}(g_{1},\ldots ,g_{m})\in M$. This is a linear combination of products

$$\begin{eqnarray}\mathop{\prod }_{i=1}^{m}Z^{\operatorname{GL}_{n}}(W_{i},W_{i}^{\prime },\unicode[STIX]{x1D6F7}_{i},s_{i})\end{eqnarray}$$

where $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{n}}}(\unicode[STIX]{x1D70B})$, $W_{i}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{n}}^{-1}}(\unicode[STIX]{x1D70B}^{\prime })$ and $\unicode[STIX]{x1D6F7}_{i}\in {\mathcal{S}}(F^{n})$. Thus,

(23)

Moreover, we have a functional equation

(24)$$\begin{eqnarray}\displaystyle & & \displaystyle Z^{M}(\widehat{W}^{M},\widehat{W}^{\prime M},\widehat{\tilde{\unicode[STIX]{x1D6F7}}}^{M},(1-s_{1},\ldots ,1-s_{m}))\nonumber\\ \displaystyle & & \displaystyle \quad =\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}^{\prime }}(-1)^{m(n-1)}\mathop{\prod }_{i=1}^{m}\unicode[STIX]{x1D6FE}(s_{i},\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70B}^{\prime },\unicode[STIX]{x1D713})Z^{M}(W,W^{\prime },\tilde{\unicode[STIX]{x1D6F7}},(s_{1},\ldots ,s_{m}))\end{eqnarray}$$

where $\widehat{W}^{M}(l)=W(\operatorname{diag}(\overbrace{w_{n},\ldots ,w_{n}}^{m})\,^{t}l^{-1})$ and

$$\begin{eqnarray}\widehat{\tilde{\unicode[STIX]{x1D6F7}}}^{M}(X_{1},\ldots ,X_{m})=\int _{(F^{n})^{m}}\unicode[STIX]{x1D6F7}(Y_{1},\ldots ,Y_{m})\unicode[STIX]{x1D713}\biggl(\mathop{\sum }_{i}\left\langle X_{i},Y_{i}\right\rangle \biggr)\,dY_{1}\cdots \,dY_{m}.\end{eqnarray}$$

5.3 Proof of the theorem

The fulcrum for Theorem 5.1 is the following proposition.

Proposition 5.2. For any $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$, $W_{\operatorname{Ze}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$ and $\unicode[STIX]{x1D6F7}\in {\mathcal{S}}(\operatorname{Mat}_{m,nm}(F))$, we have

(25)$$\begin{eqnarray}Z({\mathcal{T}}^{\unicode[STIX]{x1D713}}W_{\operatorname{Ze}},{\mathcal{T}}^{\unicode[STIX]{x1D713}^{-1}}W_{\operatorname{Ze}}^{\prime },\unicode[STIX]{x1D6F7},s)=\int _{P\backslash G}Z^{M}((W_{\operatorname{ Ze}})_{g},(W_{\operatorname{Ze}}^{\prime })_{g},\tilde{\unicode[STIX]{x1D6F7}}_{g},(s-m+1,\ldots ,s))|\det g|^{s}\,dg\end{eqnarray}$$

where $(W_{\operatorname{Ze}})_{g}=\unicode[STIX]{x1D6FF}_{P}^{-1/2}\unicode[STIX]{x1D6FF}^{\prime }W_{\operatorname{Ze}}(\cdot g)\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{M}}}(\unicode[STIX]{x1D70B}^{\otimes m})$, $(W_{\operatorname{Ze}}^{\prime })_{g}=\unicode[STIX]{x1D6FF}_{P}^{-1/2}\unicode[STIX]{x1D6FF}^{\prime }W_{\operatorname{Ze}}^{\prime }(\cdot g)\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N_{M}}^{-1}}(\unicode[STIX]{x1D70B}^{\prime \otimes m})$ and $\tilde{\unicode[STIX]{x1D6F7}}_{g}$ is given by (20). The integral on the right-hand side is absolutely convergent for $\operatorname{Re}s+\mathbf{e}(\unicode[STIX]{x1D70B})+\mathbf{e}(\unicode[STIX]{x1D70B}^{\prime })+1>m$.

Proof. Write $Z({\mathcal{T}}^{\unicode[STIX]{x1D713}}W_{\operatorname{Ze}},{\mathcal{T}}^{\unicode[STIX]{x1D713}^{-1}}W_{\operatorname{Ze}}^{\prime },\unicode[STIX]{x1D6F7},s)$ as

$$\begin{eqnarray}\int _{D\backslash G}\int _{U^{\prime }\backslash D}{\mathcal{T}}^{\unicode[STIX]{x1D713}}W_{\operatorname{ Ze}}(lg){\mathcal{T}}^{\unicode[STIX]{x1D713}^{-1}}W_{\operatorname{Ze}}^{\prime }(lg)|\det l|^{s-m}\,dl~\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{s}\,dg.\end{eqnarray}$$

By Proposition 4.1 we get

$$\begin{eqnarray}\displaystyle & & \displaystyle \int _{D\backslash G}\int _{N_{D}\backslash D}W_{\operatorname{Ze}}(lg)W_{\operatorname{Ze}}^{\prime }(lg)|\det l|^{s-m}\,dl~\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{s}\,dg\nonumber\\ \displaystyle & & \displaystyle \quad =\int _{N_{D}\backslash G}W_{\operatorname{Ze}}(g)W_{\operatorname{Ze}}^{\prime }(g)\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)\,dg|\det g|^{s}\,dg.\nonumber\end{eqnarray}$$

We write this as

$$\begin{eqnarray}\int _{P\backslash G}\int _{N_{M}\backslash M}W_{\operatorname{Ze}}(lg)W_{\operatorname{Ze}}^{\prime }(lg)\int _{U_{D}\backslash U}\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}ulg)\,du~|\det l|^{s}\unicode[STIX]{x1D6FF}_{P}(l)^{-1}\,dl~|\det g|^{s}\,dg.\end{eqnarray}$$

The required identity now follows from (19). For convergence, as in the proof of Proposition 4.1, we may assume that $\unicode[STIX]{x1D6F7}\geqslant 0$, $s\in \mathbb{R}$, $\unicode[STIX]{x1D70B}^{\prime }=\bar{\unicode[STIX]{x1D70B}}$ and $W_{2}=\overline{W_{1}}$, so that all the integrands considered above are non-negative. Therefore, the manipulations are justified for $s+2\mathbf{e}(\unicode[STIX]{x1D70B})+1>m$ by (23).◻

Proposition 5.2 immediately implies the first part of Theorem 5.1 (absolute convergence). In view of Remark 3.6, Proposition 5.2 also reduces the second and third parts of Theorem 5.1 (analyticity and unramified computation) to the analogous statements for the usual Rankin–Selberg integrals.

Remark 5.3. If $\unicode[STIX]{x1D70E}$ and $\unicode[STIX]{x1D70E}^{\prime }$ are unramified, then

$$\begin{eqnarray}\mathop{\prod }_{i=0}^{m-1}L(s-i,\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70B}^{\prime })=L\biggl(s-\frac{m-1}{2},\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70E}^{\prime }\biggr)=L\biggl(s-\frac{m-1}{2},\unicode[STIX]{x1D70E}\times \unicode[STIX]{x1D70B}^{\prime }\biggr).\end{eqnarray}$$

However, in general for $\unicode[STIX]{x1D70B},\unicode[STIX]{x1D70B}^{\prime }\in \operatorname{Irr}_{\operatorname{gen}}\operatorname{GL}_{n}$, the equality

$$\begin{eqnarray}L\biggl(s-\frac{m-1}{2},\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70E}^{\prime }\biggr)=L\biggl(s-\frac{m-1}{2},\unicode[STIX]{x1D70E}\times \unicode[STIX]{x1D70B}^{\prime }\biggr)\end{eqnarray}$$

does not always hold.

Finally, we prove the functional equation (last part of Theorem 5.1).

For any $W_{\operatorname{Ze}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}}(\unicode[STIX]{x1D70E})$, define $\widehat{W_{\operatorname{Ze}}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{N}^{-1}}(\unicode[STIX]{x1D70E}^{\vee })$ by $\widehat{W_{\operatorname{Ze}}}(g)=W_{\operatorname{Ze}}(w_{nm}\,^{t}g^{-1})$. Then $\widehat{{\mathcal{T}}W_{\operatorname{Ze}}}={\mathcal{T}}(\widehat{W_{\operatorname{Ze}}})$. Note that $w_{mn}=\operatorname{diag}(\overbrace{w_{n},\ldots ,w_{n}}^{m})w_{m,n}$ where $w_{m,n}=\unicode[STIX]{x1D704}(w_{m})$; write $g^{\prime }=w_{m,n}\,^{t}g^{-1}$, $g\in G$. Then, for any $g\in G$, we have

$$\begin{eqnarray}(\widehat{W_{\operatorname{Ze}}})_{g}(l)=\widehat{(W_{\operatorname{Ze}})_{g^{\prime }}}^{M}(w_{m,n}lw_{m,n}^{-1}),\quad l\in M,\end{eqnarray}$$

and by Fourier inversion

$$\begin{eqnarray}\widetilde{(\widehat{\unicode[STIX]{x1D6F7}})}_{g}(v_{1},\ldots ,v_{m})=|\det g|^{-m}\widehat{\tilde{\unicode[STIX]{x1D6F7}}_{g^{\prime }}}^{M}(v_{m},\ldots ,v_{1}),\quad v_{1},\ldots ,v_{m}\in F^{n}.\end{eqnarray}$$

The last part of Theorem 5.1 therefore follows from Proposition 5.2 and the functional equation (24) using the change of variable $g\mapsto g^{\prime }$ in the integral on the right-hand side of (25).

This finishes the proof of Theorem 5.1.

6.1 Relation between zeta integrals and ${\mathcal{B}}_{\operatorname{Sh}}$

Recall that $Q=D\rtimes {M^{\prime }}^{\unicode[STIX]{x1D6F9}}$, $\unicode[STIX]{x1D6FF}_{Q}|_{D}=\unicode[STIX]{x1D6FF}_{D}=|\det |^{m}$ and $\unicode[STIX]{x1D6FF}_{Q}|_{{M^{\prime }}^{\unicode[STIX]{x1D6F9}}}=1$. Hence, we can write $Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)$ as

$$\begin{eqnarray}\int _{Q\backslash G}\int _{U^{\prime }\backslash D}\int _{{M^{\prime }}^{\unicode[STIX]{x1D6F9}}}W_{\operatorname{Sh}}(lpg)W_{\operatorname{Sh}}^{\prime }(lpg)\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}lg)|\det l|^{s}\,dl~|\det p|^{s-m}\,dp~~|\det g|^{s}\,dg.\end{eqnarray}$$

Using Lemma 3.9 and the identification $\unicode[STIX]{x1D704}:\operatorname{GL}_{m}\rightarrow {M^{\prime }}^{\unicode[STIX]{x1D6F9}}$, we get

(26)$$\begin{eqnarray}Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)=\int _{Q\backslash G}{\mathcal{B}}_{\operatorname{Sh}}(W_{\operatorname{Sh}}(\cdot g),W_{\operatorname{Sh}}^{\prime }(\cdot g),s-m)f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}^{\prime }},s}(g)\,dg\end{eqnarray}$$

where, for any character $\unicode[STIX]{x1D714}$ of $F^{\ast }$,

$$\begin{eqnarray}f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714},s}(g)=\int _{\operatorname{GL}_{m}}\unicode[STIX]{x1D6F7}_{g}^{\prime }(l)\unicode[STIX]{x1D714}(\det l)|\det l|^{ns}\,dl~|\det g|^{s}\end{eqnarray}$$

and $\unicode[STIX]{x1D6F7}_{g}^{\prime }\in {\mathcal{S}}(\operatorname{Mat}_{m,m}(F))$ is given by $\unicode[STIX]{x1D6F7}_{g}^{\prime }(X)=\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D707}(X)g)$ where $\unicode[STIX]{x1D707}(X)\in \operatorname{Mat}_{m\times nm}$ is the matrix whose $i$th row is $\sum _{j=1}^{m}X_{i,j}e_{nj}$. Note that $\unicode[STIX]{x1D6F7}\mapsto f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714},s}$ is an intertwining map from ${\mathcal{S}}(\operatorname{Mat}_{m,nm}(F))\otimes |\det |^{s}$ to $\operatorname{Ind}_{Q}^{G}\unicode[STIX]{x1D708}_{s}$ where $\unicode[STIX]{x1D708}_{s}$ is the character on $Q$ such that $\unicode[STIX]{x1D708}_{s}|_{D}=|\det |^{s-m/2}$ and $\unicode[STIX]{x1D708}_{s}\circ \unicode[STIX]{x1D704}=\unicode[STIX]{x1D714}^{-1}\circ \det$.

Let $\operatorname{ord}_{{\mathcal{B}}}(s)=\operatorname{ord}_{{\mathcal{B}};\unicode[STIX]{x1D70E},\unicode[STIX]{x1D70E}^{\prime }}(s)$ be the maximal order of pole of ${\mathcal{B}}_{i}(W_{i},W_{i}^{\prime },\cdot )$ at $s$ for $i=0,\ldots ,(m-1)n$ as we vary $W_{i}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}}(\unicode[STIX]{x1D70E})$, $W_{i}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U_{i}}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$. (Recall that this does not depend on $i$ by (16).) By Corollary 3.13, we have $\operatorname{ord}_{{\mathcal{B}}}(s)\geqslant 0$ for all $s$.

Similarly, let $\operatorname{ord}_{Z}(s)=\operatorname{ord}_{Z;\unicode[STIX]{x1D70E},\unicode[STIX]{x1D70E}^{\prime }}(s)\geqslant 0$ be the maximal order of pole of $Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},\cdot )$ at $s$ as we vary $W\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$, $W^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$ and $\unicode[STIX]{x1D6F7}\in {\mathcal{S}}(\operatorname{Mat}_{m,nm}(F))$. By Lemma 6.1, we have $\operatorname{ord}_{Z}(s)\geqslant 0$ for all $s$. We can sharpen this as follows.

Proof. It is enough to prove the meromorphic continuation for $i=(m-1)n$, that is, for ${\mathcal{B}}_{\operatorname{Sh}}$. This case follows from equality (26). Indeed, taking $\unicode[STIX]{x1D714}=\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}}\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D70B}^{\prime }}$ and $\unicode[STIX]{x1D6F7}$ to be the characteristic function of a small neighborhood of $\unicode[STIX]{x1D702}$, $f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714},s}$ is supported in $Q\unicode[STIX]{x1D6FA}$ for a small neighborhood $\unicode[STIX]{x1D6FA}$ of $e$ and hence $Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)$ is a non-zero constant multiple of ${\mathcal{B}}_{\operatorname{Sh}}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },s-m)$. We also get that $\operatorname{ord}_{{\mathcal{B}}}(s-m)\leqslant \operatorname{ord}_{Z}(s)$ for all $s$.

On the other hand, $f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714},s}(g)$ is a generalized Tate integral with respect to $\operatorname{GL}_{m}$, and hence

$$\begin{eqnarray}L\biggl(ns-\frac{m-1}{2},\unicode[STIX]{x1D714}\circ \det _{\operatorname{GL}_{m}}\biggr)f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714},s}=\biggl(\mathop{\prod }_{i=0}^{m-1}L(ns-i,\unicode[STIX]{x1D714})\biggr)f_{\unicode[STIX]{x1D6F7},\unicode[STIX]{x1D714},s}\end{eqnarray}$$

is entire. We get from (26) that $\operatorname{ord}_{Z}(s)\leqslant \operatorname{ord}_{{\mathcal{B}}}(s-m)$ unless $\unicode[STIX]{x1D714}=|\cdot |^{j-ns}$ for some $j\in \{0,\ldots ,m-1\}$, in which case $\operatorname{ord}_{Z}(s)\leqslant \operatorname{ord}_{{\mathcal{B}}}(s-m)+1$. The corollary follows.◻

Remark 6.4. In general, we do not know what precisely is the fractional ideal of $\mathbb{Z}[q^{\pm s}]$ generated by

$$\begin{eqnarray}Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)\quad \text{where }W_{\operatorname{ Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E}),~W_{\operatorname{Sh}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E}^{\prime }),~\unicode[STIX]{x1D6F7}\in {\mathcal{S}}(\operatorname{Mat}_{m,nm}(F)).\end{eqnarray}$$

If both $\unicode[STIX]{x1D70B}$ and $\unicode[STIX]{x1D70B}^{\prime }$ are unitarizable, then we expect that this ideal is generated by $\prod _{i=0}^{m-1}L(s-i,\unicode[STIX]{x1D70B}\times \unicode[STIX]{x1D70B}^{\prime })$, that is, part (ii) of Theorem 5.1 is tight in this case.

6.2 More results in the (AT) case

Proposition 6.6. Suppose that $\unicode[STIX]{x1D70B}$ is (AT) and let $\unicode[STIX]{x1D70B}^{\prime }=\unicode[STIX]{x1D70B}^{\vee }$. Then, for any $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$, $W_{\operatorname{Sh}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}^{-1}}(\unicode[STIX]{x1D70E}^{\prime })$, we have

$$\begin{eqnarray}Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},m)={\mathcal{B}}_{\operatorname{ Sh}}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime })\hat{\unicode[STIX]{x1D6F7}}(0)\end{eqnarray}$$

where both sides are well defined.

Proof. By the first part of Theorem 5.1, the integral defining $Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},m)$ is absolutely convergent. Moreover, since the modulus function of $D$ is $|\det |^{m}$, we can write

$$\begin{eqnarray}\displaystyle Z(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},m) & = & \displaystyle \int _{U^{\prime }\backslash G}W_{\operatorname{Sh}}(g)W_{\operatorname{Sh}}^{\prime }(g)\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{m}\,dg\nonumber\\ \displaystyle & = & \displaystyle \int _{D\backslash G}\int _{U^{\prime }\backslash D}W_{\operatorname{Sh}}(pg)W_{\operatorname{Sh}}^{\prime }(pg)\,dp~\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{m}\,dg.\nonumber\end{eqnarray}$$

For $\unicode[STIX]{x1D70B}^{\prime }=\unicode[STIX]{x1D70B}^{\vee }$, by Theorem 4.3 we get

$$\begin{eqnarray}{\mathcal{B}}_{\operatorname{Sh}}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime })\int _{D\backslash G}~\unicode[STIX]{x1D6F7}(\unicode[STIX]{x1D702}g)|\det g|^{m}dg=\hat{\unicode[STIX]{x1D6F7}}(0){\mathcal{B}}_{\operatorname{ Sh}}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime })\end{eqnarray}$$

as required. ◻

From the functional equations (22) we deduce the following corollary.

Under mild assumptions, we can give a lower bound for the real part of the first location of a pole.

Proof. Indeed, by Lemma 6.9 we have $\operatorname{ord}_{Z}(s)>0$ for some $s$ with $\operatorname{Re}s\geqslant m-1$. Hence, by Proposition 6.2, $\operatorname{ord}_{{\mathcal{B}}}(s-m)>0$ for that $s$ (since $n,m>1$). Therefore, the integral defining ${\mathcal{B}}_{\operatorname{Sh}}(W_{\operatorname{Sh}},\overline{W_{\operatorname{Sh}}},s)$ diverges for all $s\leqslant -1$ (cf. (15)). In particular,

$$\begin{eqnarray}\int _{U^{\prime }{M^{\prime }}^{\unicode[STIX]{x1D6F9}}\backslash G}|W_{\operatorname{Sh}}(g)|^{2}\,dg=\int _{Q\backslash G}\int _{U^{\prime }\backslash D}|W_{\operatorname{Sh}}(g)|^{2}|\det g|^{-m}\,dg\end{eqnarray}$$

diverges. ◻

Our final result in this section is the following lemma.

Lemma 6.11. Suppose that $\unicode[STIX]{x1D70B}$ is (AT) and let $\unicode[STIX]{x1D70B}^{\prime }=\unicode[STIX]{x1D70B}^{\vee }$. Then $\operatorname{ord}_{Z}(0)=\operatorname{ord}_{{\mathcal{B}}}(-m)+1$. In particular, if $\unicode[STIX]{x1D70B}$ is supercuspidal, then ${\mathcal{B}}$ is holomorphic at $s=-m$ (cf. Example 6.8). Moreover, let

$$\begin{eqnarray}Z^{\ast }(W_{\operatorname{ Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7})=\lim _{s\rightarrow 0}(q^{s}-1)^{\operatorname{ord}_{Z}(0)}Z(W_{\operatorname{ Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7},s)\end{eqnarray}$$

and

$$\begin{eqnarray}{\mathcal{B}}_{\operatorname{Sh}}^{\ast }(W_{\operatorname{ Sh}},W_{\operatorname{Sh}}^{\prime })=\lim _{s\rightarrow -m}(q^{s+m}-1)^{\operatorname{ord}_{{\mathcal{B}}}(-m)}{\mathcal{B}}_{\operatorname{Sh}}(W_{\operatorname{Sh}},W_{\operatorname{Sh}}^{\prime },s).\end{eqnarray}$$

Then there exists a constant $c$ (depending only on $F$, $m$ and $n$) such that

$$\begin{eqnarray}Z^{\ast }(W_{\operatorname{ Sh}},W_{\operatorname{Sh}}^{\prime },\unicode[STIX]{x1D6F7})=c\unicode[STIX]{x1D6F7}(0)\int _{Q\backslash G}{\mathcal{B}}_{\operatorname{Sh}}^{\ast }(W_{\operatorname{ Sh}}(\cdot g),W_{\operatorname{Sh}}^{\prime }(\cdot g),-m)\,dg\end{eqnarray}$$

for all $W_{\operatorname{Sh}}\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}}(\unicode[STIX]{x1D70E})$, $W_{\operatorname{Sh}}^{\prime }\in {\mathcal{W}}^{\unicode[STIX]{x1D6F9}_{U^{\prime }}^{-1}}(\unicode[STIX]{x1D70E}^{\vee })$ and $\unicode[STIX]{x1D6F7}\in {\mathcal{S}}(\operatorname{Mat}_{m,nm}(F))$.

We may view

$$\begin{eqnarray}\int _{Q\backslash G}{\mathcal{B}}_{\operatorname{Sh}}^{\ast }(W_{\operatorname{ Sh}}(\cdot g),W_{\operatorname{Sh}}^{\prime }(\cdot g),-m)\,dg\end{eqnarray}$$

as a regularization of

$$\begin{eqnarray}\int _{{M^{\prime }}^{\unicode[STIX]{x1D6F9}}U^{\prime }\backslash G}W_{\operatorname{Sh}}(g)W_{\operatorname{Sh}}^{\prime }(g)\,dg.\end{eqnarray}$$

(Recall that the latter diverges for $W_{\operatorname{Sh}}^{\prime }=\overline{W_{\operatorname{Sh}}}$ if $m>1$.)