1.1 General results

Let $\mathbb{F}$ be a local field of characteristic zero, $G$ a reductive group defined over $\mathbb{F}$ , $\mathfrak{g}$ its Lie algebra and $\mathfrak{g}^{\ast }$ the dual space to $\mathfrak{g}$ . A Whittaker pair is an ordered pair $(S,\unicode[STIX]{x1D711})\in \mathfrak{g}\times \mathfrak{g}^{\ast }$ such that $S$ is semi-simple with eigenvalues of the adjoint action $\text{ad}(S)$ lying in $\mathbb{Q}$ , and $\text{ad}^{\ast }(S)(\unicode[STIX]{x1D711})=-2\unicode[STIX]{x1D711}$ . Note that $\unicode[STIX]{x1D711}$ is necessarily nilpotent and given by the Killing form pairing with a (unique) nilpotent element $f=f_{\unicode[STIX]{x1D711}}\in \mathfrak{g}$ . Following [Reference Moeglin and WaldspurgerMW87] we attach to $(S,\unicode[STIX]{x1D711})$ a certain smooth representation ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ of $G$ called a degenerate Whittaker model for $G$ .

Two classes of Whittaker pairs and the corresponding models will be of special interest to us. If $S$ is a neutral element for $f_{\unicode[STIX]{x1D711}}$ (see Definition 2.2.2 below), then we will say that $(S,\unicode[STIX]{x1D711})$ is a neutral pair and call ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ a neutral model or a generalized model (see [Reference KawanakaKaw85, Reference Moeglin and WaldspurgerMW87, Reference YamashitaYam86, Reference Gomez and ZhuGZ14]). The second class consists of Whittaker pairs $(S,\unicode[STIX]{x1D711})$ where $S$ is the neutral element of a principal $\mathfrak{sl}_{2}$ -triple in $G$ ; in this case $f_{\unicode[STIX]{x1D711}}$ is necessarily a principal nilpotent element for a Levi subgroup of $G$ , and we will say $(S,\unicode[STIX]{x1D711})$ is a PL pair, and ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ is a PL model or a principal degenerate model (see [Reference ZelevinskyZel80, Reference Moeglin and WaldspurgerMW87, Reference Bushnell and HenniartBH03, Reference Gourevitch and SahiGS13]).

We will now sketch the definition of ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ , referring to § 2.5 below for more details. Let $\mathfrak{u}\subset \mathfrak{g}$ denote the sum of all eigenspaces of $\text{ad}(S)$ with eigenvalues at least 1. Note that $\mathfrak{u}$ is a nilpotent subalgebra and let $U:=\text{Exp}(\mathfrak{u})\subset G$ be the corresponding nilpotent subgroup. Fix an additive character of $\mathbb{F}$ . Suppose first that 1 is not an eigenvalue of $\text{ad}(S)$ . Then the restriction of $\unicode[STIX]{x1D711}$ to $\mathfrak{u}$ is a character of $\mathfrak{u}$ , which defines a character $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ of $U$ . The degenerate Whittaker model is defined to be the Schwartz induction of this character: ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}:=\text{ind}_{U}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ . If 1 is an eigenvalue of $\text{ad}(S)$ , then consider the anti-symmetric form on $\mathfrak{u}$ given by $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}}(X,Y):=\unicode[STIX]{x1D711}([X,Y])$ and let $\mathfrak{n}$ denote the radical of this form. Let $\mathfrak{n}^{\prime }:=\mathfrak{n}\cap \text{Ker}\,\unicode[STIX]{x1D711}$ , and let $N^{\prime }:=\text{Exp}(\mathfrak{n}^{\prime })$ . It is easy to show that $N^{\prime }$ is a normal subgroup of $U$ and $U/N^{\prime }$ is isomorphic to a (generalized) Heisenberg group, of which $\unicode[STIX]{x1D711}$ defines a central character $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ . Let $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}}$ denote the oscillator representation of $U/N^{\prime }$ with central character $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ . Consider $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}}$ as a representation of $U$ and define ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}:=\text{ind}_{U}^{G}\,\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}}$ .

If $(S,\unicode[STIX]{x1D711})$ is a neutral pair, then the generalized Whittaker model ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ does not depend on the choice of a neutral $S$ , and will thus be denoted ${\mathcal{W}}_{\unicode[STIX]{x1D711}}$ . Since conjugate nilpotent elements give rise to isomorphic generalized Whittaker models, we will also use the notation ${\mathcal{W}}_{{\mathcal{O}}}$ for a nilpotent coadjoint orbit ${\mathcal{O}}$ .

We denote by $\overline{G_{S}\tilde{\unicode[STIX]{x1D711}}}\subset \mathfrak{g}^{\ast }$ the closure of the orbit of $\tilde{\unicode[STIX]{x1D711}}$ under the coadjoint action of the centralizer of $S$ in $G$ .

In particular, taking $\tilde{\unicode[STIX]{x1D711}}=\unicode[STIX]{x1D711}$ we see that the generalized Whittaker model maps onto any degenerate Whittaker model corresponding to the same $\unicode[STIX]{x1D711}$ . In fact, we prove a more general result (Theorem 3.0.1) on epimorphisms between pairs of degenerate Whittaker models, which enables one to define a preorder on the pairs $(S,\unicode[STIX]{x1D711})$ . If we fix $f_{\unicode[STIX]{x1D711}}$ to be a regular nilpotent element for a Levi subgroup of $G$ , then the minimal elements under this preorder are the PL Whittaker pairs (see § 3.3). The corresponding principal degenerate Whittaker models are inductions of (possibly degenerate) characters of the nilradicals of minimal parabolic subgroups.

The above results have applications to the study of Whittaker functionals on representations of $G$ . Following [Reference Gomez and ZhuGZ14] we briefly recall the necessary background.

Let ${\mathcal{M}}(G)$ denote the category of smooth admissibleFootnote ¹ (finitely generated) representations of $G$ (see [Reference Bernstein and ZelevinskyBZ76, Reference CasselmanCas89, Reference WallachWal92]). For $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ and a nilpotent orbit ${\mathcal{O}}\subset \mathfrak{g}$ denote

(1) $$\begin{eqnarray}{\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}):=\text{Hom}_{G}({\mathcal{W}}_{{\mathcal{O}}},\unicode[STIX]{x1D70B}^{\ast }).\end{eqnarray}$$

The study of Whittaker and generalized Whittaker models for representations of reductive groups over local fields evolved in connection with the theory of automorphic forms (via their Fourier coefficients), and has found important applications in both areas. See for example [Reference ShalikaSha74, Reference Novodvorskii and Piatetski-ShapiroNP73, Reference KostantKos78, Reference KawanakaKaw85, Reference YamashitaYam86, Reference WallachWal88, Reference GinzburgGin06, Reference JiangJia07, Reference Ginzburg, Rallis and SoudryGRS11]. From the point of view of representation theory, the space of generalized Whittaker models may be viewed as one kind of nilpotent invariant associated to smooth representations. Another important invariant is the wave-front cycle:

(2) $$\begin{eqnarray}\operatorname{WFC}(\unicode[STIX]{x1D70B})=\mathop{\sum }_{\stackrel{{\mathcal{O}}\subset \mathfrak{g}^{\ast }}{\text{nilpotent}}}c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})[{\mathcal{O}}],\end{eqnarray}$$

defined by Harish-Chandra in the non-archimedean case and by Howe and Barbasch–Vogan in the archimedean case ([Reference HoweHow81, Reference Barbasch and VoganBV80]; see also [Reference RossmannRos95, Reference Schmid and VilonenSV00]). Recently, the behavior of the wave-front set and the generalized Whittaker models under $\unicode[STIX]{x1D703}$ -correspondence was studied in [Reference Gomez and ZhuGZ14, Reference Loke and MaLM15b].

For $\mathbb{F}$ non-archimedean, Mœglin and Waldspurger [Reference Moeglin and WaldspurgerMW87] have established that $\operatorname{WFC}(\unicode[STIX]{x1D70B})$ completely controls the spaces of generalized Whittaker models of interest, namely, the set of maximal orbits in $\operatorname{WFC}(\unicode[STIX]{x1D70B})$ coincides with the set of maximal orbits such that ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})\neq 0$ , and for any orbit ${\mathcal{O}}$ in this set we have

$$\begin{eqnarray}c_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})=\dim {\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B}).\end{eqnarray}$$

In [Reference Moeglin and WaldspurgerMW87] it is assumed that the residue characteristic is odd. This assumption was recently removed in [Reference VarmaVar14]. In [Reference MoeglinMoe96], the main result of [Reference Moeglin and WaldspurgerMW87] is used in order to prove that for classical groups, the maximal orbits in the wave-front set of a tempered representation are distinguished, and the maximal orbits in the wave-front set of any admissible representation are special. The latter was recently generalized in [Reference Jiang, Liu and SavinJLS14]. Partial analogs of these results hold also for archimedean $\mathbb{F}$ , see [Reference HarrisHar12] for the former and [Reference Barbasch and VoganBV82, Reference Barbasch and VoganBV83, Reference JosephJos80] for the latter.

For archimedean $\mathbb{F}$ , the correspondence between the wave-front set and non-vanishing of degenerate Whittaker models is not yet (fully) understood, except for several special cases including the representations with the largest Gelfand–Kirillov dimension [Reference VoganVog78, Reference MatumotoMat92] and unitary highest weight modules [Reference YamashitaYam01]. For the latter, the wave-front set was computed earlier in [Reference PrzebindaPrz91]. In [Reference MatumotoMat87], it is shown in full generality that every orbit ${\mathcal{O}}$ with ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})\neq 0$ lies in the Zariski closure of some orbit in $\operatorname{WFC}(\unicode[STIX]{x1D70B})$ . The paper [Reference Gourevitch and SahiGS15] proves an expected existence of non-zero maps from principal degenerate Whittaker models to admissible representations, and by Proposition 3.3.4 below, any other degenerate Whittaker model corresponding to the same orbit is mapped onto a principal degenerate Whittaker model. Let $\operatorname{WF}(\unicode[STIX]{x1D70B})$ denote the closure in the local field topology of the union of all of the orbits in $\operatorname{WFC}(\unicode[STIX]{x1D70B})$ . In § 3.3 we review the results of [Reference MatumotoMat87, Reference Gourevitch and SahiGS15] and deduce, using Theorem A, the following theorem.

Theorem B (§ 3.3).

Let $G$ be a complex reductive group and let $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ . Let $(S,\unicode[STIX]{x1D711})$ be a Whittaker pair such that $\unicode[STIX]{x1D711}$ is given by Killing form pairing with a principal nilpotent element of the Lie algebra of a Levi subgroup of $G$ . Then

$$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\neq 0\Leftrightarrow \unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B}).\end{eqnarray}$$

If $G$ is classical, then the set $\operatorname{WF}(\unicode[STIX]{x1D70B})$ is uniquely determined by its intersection with the set of ‘principal Levi’ nilpotents $\unicode[STIX]{x1D711}$ ; see [Reference Gourevitch and SahiGS15, Theorem D]. A result related to Theorem B is proved by Matumoto in [Reference MatumotoMat90]. Namely, let $G$ be a complex reductive group and let $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ have regular infinitesimal character. Let $(S,\unicode[STIX]{x1D711})$ be a Whittaker pair and assume that the orbit of $\unicode[STIX]{x1D711}$ is dense in $\operatorname{WF}(\unicode[STIX]{x1D70B})$ , and also that it contains a dense subset of the nilradical of the parabolic subgroup defined by $S$ . Then $0<\dim {\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})<\infty$ . Matumoto also proves the vanishing of the corresponding higher homology groups.

For $\mathbb{F}=\mathbb{R}$ a weaker version of Theorem B holds, see Corollary 3.3.7 below.

In § 5 we give choice-free definitions of degenerate Whittaker models. These definitions use the Deligne filtration instead of the Jacobson–Morozov theorem and thus might be suitable for local fields of positive characteristic.

In the global case, instead of degenerate Whittaker models one considers explicit functionals on automorphic representations defined by integration against a character of a nilpotent subgroup. Such functionals are called Whittaker–Fourier coefficients and denoted ${\mathcal{W}}{\mathcal{F}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})$ . In § 6 we give the definitions and explain how to adapt our arguments to the global case and deduce the following theorem.

1.2 Further results for $\text{GL}_{n}(\mathbb{F})$

For $G_{n}:=\text{GL}_{n}(\mathbb{F})$ we show that Theorem A allows one to compare degenerate Whittaker models corresponding to any nilpotent orbits ${\mathcal{O}},{\mathcal{O}}^{\prime }\subset \mathfrak{g}_{n}^{\ast }:=\mathfrak{gl}(n,\mathbb{F})^{\ast }$ such that ${\mathcal{O}}\subset \overline{{\mathcal{O}}^{\prime }}$ . To that end we prove the following geometric theorem.

Using Theorem B [Reference MatumotoMat87] and Corollary 3.3.7 for archimedean $\mathbb{F}$ , and Theorems A and D, together with the results of [Reference Moeglin and WaldspurgerMW87, Reference VarmaVar14] for non-archimedean $\mathbb{F}$ we deduce the following theorem.

Theorem E (§ 4.3).

Let $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G_{n})$ . Let ${\mathcal{O}}\subset \mathfrak{g}_{n}^{\ast }$ be a nilpotent orbit. Then we have

(3) $$\begin{eqnarray}{\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})\neq 0\Leftrightarrow {\mathcal{O}}\subset \operatorname{WF}(\unicode[STIX]{x1D70B}).\end{eqnarray}$$

In § 4.4 we give a precise description of the generalized Whittaker spaces in terms of certain functors $E^{k}$ introduced in [Reference Aizenbud, Gourevitch and SahiAGS15a, Reference Aizenbud, Gourevitch and SahiAGS15b] in connection with the generalization of the theory of Bernstein–Zelevinsky derivatives to the archimedean setting. We will also use related functors $I^{k}$ that go in the other direction. We refer to § 4.4 below for the precise definitions of both functors.

Theorem F (§ 4.4).

Let $\unicode[STIX]{x1D706}=(\unicode[STIX]{x1D706}_{1},\ldots ,\unicode[STIX]{x1D706}_{k})$ be a partition of $n$ and ${\mathcal{O}}_{\unicode[STIX]{x1D706}}\subset \mathfrak{g}_{n}^{\ast }$ be the corresponding nilpotent orbit. Then

(4) $$\begin{eqnarray}{\mathcal{W}}_{{\mathcal{O}}_{\unicode[STIX]{x1D706}}}\simeq I^{\unicode[STIX]{x1D706}_{1}}(\cdots I^{\unicode[STIX]{x1D706}_{k}}(\mathbb{C})\cdots \,),\end{eqnarray}$$

where $\mathbb{C}$ denotes the one-dimensional representation of the trivial group $G_{0}$ , and for $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G_{n})$ we have

(5) $$\begin{eqnarray}{\mathcal{W}}_{{\mathcal{O}}_{\unicode[STIX]{x1D706}}}(\unicode[STIX]{x1D70B})\simeq (E^{\unicode[STIX]{x1D706}_{k}}(\cdots E^{\unicode[STIX]{x1D706}_{1}}(\unicode[STIX]{x1D70B})\cdots \,))^{\ast }.\end{eqnarray}$$

Let ${\mathcal{O}}\subset \mathfrak{g}_{n}^{\ast }$ be a nilpotent orbit. Let ${\mathcal{M}}^{{\leqslant}{\mathcal{O}}}(G_{n})$ denote the Serre subcategory of ${\mathcal{M}}(G_{n})$ consisting of representations with wave-front set inside the closure of ${\mathcal{O}}$ and let ${\mathcal{M}}^{{<}{\mathcal{O}}}(G_{n})$ denote the Serre subcategory of ${\mathcal{M}}^{{\leqslant}{\mathcal{O}}}(G_{n})$ consisting of representations with wave-front set not containing ${\mathcal{O}}$ . Let ${\mathcal{M}}^{{\mathcal{O}}}(G_{n}):={\mathcal{M}}^{{\leqslant}{\mathcal{O}}}(G_{n})/{\mathcal{M}}^{{<}{\mathcal{O}}}(G_{n})$ denote the quotient category.

Using the main results of [Reference Aizenbud, Gourevitch and SahiAGS15a, Reference Aizenbud, Gourevitch and SahiAGS15b, Reference Gourevitch and SahiGS13, Reference Gourevitch and SahiGS15] we obtain the following corollary.

This corollary is new only in the archimedean case, since over $p$ -adic fields exactness is well-known and finiteness of dimension is shown in [Reference Moeglin and WaldspurgerMW87]. It is also shown in [Reference Moeglin and WaldspurgerMW87] that ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})$ is one-dimensional for all irreducible $\unicode[STIX]{x1D70B}\in {\mathcal{M}}^{{\mathcal{O}}}(G_{n})$ . Over archimedean fields, ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})$ is clearly not one-dimensional for any irreducible $\unicode[STIX]{x1D70B}$ of finite dimension bigger than one.

Let us also comment that for any irreducible $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G_{n})$ , $\operatorname{WF}(\unicode[STIX]{x1D70B})$ is the closure of a single nilpotent orbit. Over archimedean fields this follows from [Reference Borho and BrylinskiBB82, Reference JosephJos85] and over non-archimedean fields this is proven in [Reference Moeglin and WaldspurgerMW87]. More generally, it follows from [Reference JosephJos85] that for any real reductive group $G$ and any irreducible $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ , all of the maximal orbits in $\operatorname{WF}(\unicode[STIX]{x1D70B})$ lie in a single complex nilpotent orbit. An analogous statement is conjectured over $p$ -adic fields but not proven yet.

We conjecture that the functor $\unicode[STIX]{x1D70B}\mapsto {\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})$ is exact for all reductive groups, and we hope to prove this in the future, generalizing the technique of [Reference Aizenbud, Gourevitch and SahiAGS15b].

1.3 The structure of our proofs

Let us first describe the idea of the proof of Theorem A in the case $\unicode[STIX]{x1D711}=\tilde{\unicode[STIX]{x1D711}}$ . We first show that $S$ can be presented as $h+Z$ , where $h$ is a neutral element for $\unicode[STIX]{x1D711}$ and $Z$ commutes with $h$ and $\unicode[STIX]{x1D711}$ . Then we consider a deformation $S_{t}=h+tZ$ , and denote by $\mathfrak{u}_{t}$ the sum of eigenspaces of $\text{ad}(S_{t})$ with eigenvalues at least 1. We call a rational number $0<t<1$ regular if $\mathfrak{u}_{t}=\mathfrak{u}_{t+\unicode[STIX]{x1D700}}$ for any small enough rational $\unicode[STIX]{x1D700}$ , and critical otherwise. Note that there are finitely many critical numbers, and denote them by $t_{1}<\cdots <t_{n}$ . Denote also $t_{0}:=0$ and $t_{n+1}:=1$ . For each $t$ we define two subalgebras $\mathfrak{l}_{t},\mathfrak{r}_{t}\subset \mathfrak{u}_{t}$ . Both $\mathfrak{l}_{t}$ and $\mathfrak{r}_{t}$ are maximal isotropic subspaces with respect to the form $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}}$ , $\mathfrak{r}_{t}$ contains all of the eigenspaces of $Z$ in $\mathfrak{u}_{t}$ with positive eigenvalues and $\mathfrak{l}_{t}$ contains all of the eigenspaces with negative eigenvalues. Note that the restrictions of $\unicode[STIX]{x1D711}$ to $\mathfrak{l}_{t}$ and $\mathfrak{r}_{t}$ define characters of these subalgebras. Let $L_{t}:=\text{Exp}(\mathfrak{l}_{t})$ and $R_{t}:=\text{Exp}(\mathfrak{r}_{t})$ denote the corresponding subgroups and $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ denote their characters defined by $\unicode[STIX]{x1D711}$ . The Stone–von Neumann theorem implies

$$\begin{eqnarray}{\mathcal{W}}_{S_{t},\unicode[STIX]{x1D711}}\simeq \text{ind}_{L_{t}}^{G}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})\simeq \text{ind}_{R_{t}}^{G}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}).\end{eqnarray}$$

We show that for any $0\leqslant i\leqslant n,\,\mathfrak{r}_{t_{i}}\subset \mathfrak{l}_{t_{i+1}}$ . This gives a natural epimorphism

$$\begin{eqnarray}{\mathcal{W}}_{S_{t_{i}},\unicode[STIX]{x1D711}}\simeq \text{ind}_{L_{t_{i}}}^{G}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})\,{\twoheadrightarrow}\,\text{ind}_{R_{t_{i}}}^{G}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})\simeq {\mathcal{W}}_{S_{t_{i+1}},\unicode[STIX]{x1D711}}.\end{eqnarray}$$

Altogether, we get

$$\begin{eqnarray}{\mathcal{W}}_{h,\unicode[STIX]{x1D711}}={\mathcal{W}}_{S_{t_{0}},\unicode[STIX]{x1D711}}\,{\twoheadrightarrow}\,{\mathcal{W}}_{S_{t_{1}},\unicode[STIX]{x1D711}}{\twoheadrightarrow}\cdots \,{\twoheadrightarrow}\,{\mathcal{W}}_{S_{t_{n+1}},\unicode[STIX]{x1D711}}={\mathcal{W}}_{S,\unicode[STIX]{x1D711}}.\end{eqnarray}$$

If $\unicode[STIX]{x1D711}\neq \tilde{\unicode[STIX]{x1D711}}$ , we identify $\mathfrak{g}\simeq \mathfrak{g}^{\ast }$ using a non-degenerate invariant form and complete $\unicode[STIX]{x1D711}$ to an $sl_{2}$ -triple $(e,h,\unicode[STIX]{x1D711})$ such that $h$ commutes with $S$ . Then we show, using the Slodowy slice, that the conditions imply that $\tilde{\unicode[STIX]{x1D711}}$ is conjugate under $G_{S}$ to $\unicode[STIX]{x1D711}+\unicode[STIX]{x1D711}^{\prime }$ with $\text{ad}^{\ast }(e)(\unicode[STIX]{x1D711}^{\prime })=0$ . We finish the proof by a deformation argument similar to the case $\unicode[STIX]{x1D711}=\tilde{\unicode[STIX]{x1D711}}$ .

For Theorem B, the implication ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\neq 0\Rightarrow \unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B})$ follows from [Reference MatumotoMat87]. To prove the other direction, we show that ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ maps onto some principal degenerate Whittaker model ${\mathcal{W}}_{\tilde{S},\unicode[STIX]{x1D711}}$ . Thus the theorem follows from the non-vanishing of ${\mathcal{W}}_{\tilde{S},\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})$ (under the condition $\unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B})$ ), which was shown in [Reference Gourevitch and SahiGS15]. The same argument gives Corollary 3.3.7: an analogous statement for quasisplit real reductive groups.

For the proof of Theorem D we identify $\mathfrak{g}_{n}^{\ast }$ with $\mathfrak{g}_{n}$ using the trace form, and parameterize nilpotent orbits by partitions. Then we prove the theorem for partitions of length two by an elementary matrix conjugation argument. We finish the proof by induction. The induction argument, however, is not so easy since the statement is not ‘transitive’. For any pair of partitions $\unicode[STIX]{x1D706}\leqslant \unicode[STIX]{x1D707}$ (where ${\leqslant}$ refers to the natural order on partitions which corresponds to the closure order on orbits), we consider two pairs of partitions of two smaller numbers that add up to a number bigger than $n$ . Then we take $S^{\prime }$ and $S^{\prime \prime }$ corresponding to the two pairs of partitions and force them to coincide on the joint block by adding a scalar matrix to one of them. In this way we obtain a diagonal matrix $S\in \mathfrak{gl}_{n}(\mathbb{Z})$ that satisfies the requirements of the theorem.

For archimedean $\mathbb{F}$ , Theorem E follows from Theorem B [Reference MatumotoMat87] and Corollary 3.3.7. For non-archimedean $\mathbb{F}$ , it was shown in [Reference Moeglin and WaldspurgerMW87] that ${\mathcal{W}}_{S,\tilde{\unicode[STIX]{x1D711}}}(\unicode[STIX]{x1D70B})\neq 0$ for any Whittaker pair $(S,\tilde{\unicode[STIX]{x1D711}})$ such that the orbit of $\tilde{\unicode[STIX]{x1D711}}$ is a maximal orbit in $\operatorname{WFC}(\unicode[STIX]{x1D70B})$ . If ${\mathcal{O}}\subset \operatorname{WF}(\unicode[STIX]{x1D70B})$ , then ${\mathcal{O}}\subset \overline{{\mathcal{O}}^{\prime }}$ for some maximal orbit ${\mathcal{O}}^{\prime }\in \operatorname{WFC}(\unicode[STIX]{x1D70B})$ . Pick $(S,\tilde{\unicode[STIX]{x1D711}})$ that correspond to ${\mathcal{O}},{\mathcal{O}}^{\prime }$ by Theorem D. Then, by Theorem A, ${\mathcal{W}}_{S,\tilde{\unicode[STIX]{x1D711}}}(\unicode[STIX]{x1D70B})$ embeds into ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})$ and thus ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})\neq 0$ .

We prove Theorem F by induction on $k$ . We let $\unicode[STIX]{x1D706}^{\prime }:=(\unicode[STIX]{x1D706}_{2},\ldots ,\unicode[STIX]{x1D706}_{k})$ , and by the induction hypothesis obtain

$$\begin{eqnarray}{\mathcal{W}}_{{\mathcal{O}}_{\unicode[STIX]{x1D706}^{\prime }}}\simeq I^{\unicode[STIX]{x1D706}_{2}}(\cdots I^{\unicode[STIX]{x1D706}_{k}}(\mathbb{C})\cdots \,).\end{eqnarray}$$

Thus, in order to prove (4) we have to show that

(6) $$\begin{eqnarray}{\mathcal{W}}_{{\mathcal{O}}_{\unicode[STIX]{x1D706}}}\simeq I^{\unicode[STIX]{x1D706}_{1}}({\mathcal{W}}_{{\mathcal{O}}_{\unicode[STIX]{x1D706}}}).\end{eqnarray}$$

We note that both sides of the formula are isomorphic to inductions of the same character from two nilpotent subgroups that differ only in the last $\unicode[STIX]{x1D706}_{1}$ columns. Then we prove (6) by a deformation argument similar to the proof of Theorem A. Finally, (5) follows from (4) by a version of Frobenius reciprocity.

Corollary G follows from (5) using the properties of archimedean prederivatives proven in [Reference Aizenbud, Gourevitch and SahiAGS15a, Reference Aizenbud, Gourevitch and SahiAGS15b, Reference Gourevitch and SahiGS13, Reference Gourevitch and SahiGS15].

The proof of Theorem C is analogous to the proofs of Theorems A and D. The only difference is that we cannot apply the Stone–von Neumann theorem since in the global case we consider Whittaker–Fourier coefficients, that are some explicit functionals on an automorphic representation. We replace it by Lemma 6.0.2, that is proven by an explicit integral transform followed by a Fourier transform on a compact abelian group. This lemma is in the spirit of [Reference Ginzburg, Rallis and SoudryGRS11, Propositions 7.2 and 7.3].

If $\mathbb{F}$ is archimedean, one can consider more general models, and analogs of Theorems A–F will remain valid for them, see Remark 2.5.4.

2.1 Notation

For a semi-simple element $S$ and a rational number $r$ we denote by $\mathfrak{g}_{r}^{S}$ the $r$ -eigenspace of the adjoint action of $S$ and by $\mathfrak{g}_{{\geqslant}r}^{S}$ the sum $\bigoplus _{r^{\prime }\geqslant r}\mathfrak{g}_{r^{\prime }}^{S}$ . We will also use the notation $(\mathfrak{g}^{\ast })_{r}^{S}$ and $(\mathfrak{g}^{\ast })_{{\geqslant}r}^{S}$ for the corresponding grading and filtration of the dual Lie algebra $\mathfrak{g}^{\ast }$ . For $X\in \mathfrak{g}$ or $X\in \mathfrak{g}^{\ast }$ we denote by $\mathfrak{g}_{X}$ the centralizer of $X$ in $\mathfrak{g}$ , and by $G_{X}$ the centralizer of $X$ in $G$ . We say that an element $h\in \mathfrak{g}$ is rational semi-simple if its adjoint action on $\mathfrak{g}$ is diagonalizable with eigenvalues in $\mathbb{Q}$ .

If $(f,h,e)$ is an $\mathfrak{sl}_{2}$ -triple, we will say that $e$ is a nil-positive element for $h$ , $f$ is a nil-negative element for $h$ and $h$ is a neutral element for $e$ . For a representation $V$ of $(f,h,e)$ we denote by $V^{e}$ the space spanned by the highest-weight vectors and by $V^{f}$ the space spanned by the lowest-weight vectors.

From now on we fix a non-trivial unitary additive character

(7) $$\begin{eqnarray}\unicode[STIX]{x1D712}:\mathbb{F}\rightarrow \text{S}^{1}\end{eqnarray}$$

such that if $\mathbb{F}$ is archimedean we have $\unicode[STIX]{x1D712}(x)=\exp (2\unicode[STIX]{x1D70B}i\Re (x))$ and if $\mathbb{F}$ is non-archimedean the kernel of $\unicode[STIX]{x1D712}$ is the ring of integers.

2.2 $\mathfrak{sl}_{2}$ -triples

We will need the following lemma which summarizes several well-known facts about $\mathfrak{sl}_{2}$ -triples.

It is easy to see that the lemma continues to hold true if we replace the nil-positive elements by nil-negative ones (and $\mathfrak{g}_{2}^{h}$ by $\mathfrak{g}_{-2}^{h}$ ).

2.3 Schwartz induction

Definition 2.3.1. If $G$ is an $l$ -group, we denote by $\text{Rep}^{\infty }(G)$ the category of smooth representations of $G$ . If $H\subset G$ is a closed subgroup and $\unicode[STIX]{x1D70B}\in \text{Rep}^{\infty }(H)$ is a smooth representation of $G$ , we denote by $\text{ind}_{H}^{G}(\unicode[STIX]{x1D70B})$ the smooth compactly supported induction as in [Reference Bernstein and ZelevinskyBZ76, § 2.22].

If $G$ is an affine real algebraic group, we denote by $\text{Rep}^{\infty }(G)$ the category of smooth nuclear Fréchet representations of $G$ of moderate growth. This is essentially the same definition as in [Reference du ClouxdCl91, § 1.4] with the additional assumption that the representation spaces are nuclear (see e.g. [Reference TrevesTre67, § 50]). If $H\subset G$ is a Zariski closed subgroup, and $\unicode[STIX]{x1D70B}\in \text{Rep}^{\infty }(H)$ we denote by $\text{ind}_{H}^{G}(\unicode[STIX]{x1D70B})$ the Schwartz induction as in [Reference du ClouxdCl91, § 2]. More precisely, in [Reference du ClouxdCl91] du Cloux defines a map from the space ${\mathcal{S}}(G,\unicode[STIX]{x1D70B})$ of Schwartz functions from $G$ to the underlying space of $\unicode[STIX]{x1D70B}$ to the space $C^{\infty }(G,\unicode[STIX]{x1D70B})$ of all smooth $\unicode[STIX]{x1D70B}$ -valued functions on $G$ by $f\mapsto \overline{f}$ where

$$\begin{eqnarray}\overline{f}(x)=\int _{h\in H}\unicode[STIX]{x1D70B}(h)f(xh)\,dh,\end{eqnarray}$$

and $dh$ denotes a fixed left-invariant measure on $H$ . The Schwartz induction $\text{ind}_{H}^{G}(\unicode[STIX]{x1D70B})$ is defined to be the image of this map.

From now until the end of the subsection let $G$ be either an $l$ -group or an affine real algebraic group, and $H^{\prime }\subset H\subset G$ be (Zariski) closed subgroups.

Lemma 2.3.2 ([Reference Bernstein and ZelevinskyBZ76, Proposition 2.25(b)] and [Reference du ClouxdCl91, Lemma 2.1.6]).

For any $\unicode[STIX]{x1D70B}\in \text{Rep}^{\infty }(H^{\prime })$ we have

$$\begin{eqnarray}\text{ind}_{H^{\prime }}^{G}(\unicode[STIX]{x1D70B})\simeq \text{ind}_{H}^{G}\,\text{ind}_{H^{\prime }}^{H}(\unicode[STIX]{x1D70B}).\end{eqnarray}$$

Lemma 2.3.4. Let $\unicode[STIX]{x1D70C}\in \text{Rep}^{\infty }(H)$ , $\unicode[STIX]{x1D70B}\in \text{Rep}^{\infty }(G)$ and let $\unicode[STIX]{x1D70B}^{\ast }$ denote the dual representation, (endowed with the strong dual topology in the archimedean case). Then

$$\begin{eqnarray}\text{Hom}_{G}(\text{ind}_{H}^{G}(\unicode[STIX]{x1D70C}),\unicode[STIX]{x1D70B}^{\ast })\cong \text{Hom}_{H}(\unicode[STIX]{x1D70C},\unicode[STIX]{x1D70B}^{\ast }\unicode[STIX]{x1D6E5}_{H}^{-1}\unicode[STIX]{x1D6E5}_{G}),\end{eqnarray}$$

where $\unicode[STIX]{x1D6E5}_{H}$ and $\unicode[STIX]{x1D6E5}_{G}$ denote the modular functions of $H$ and $G$ .

The non-archimedean case of this lemma follows from [Reference Bernstein and ZelevinskyBZ76, Proposition 2.29]. We prove the archimedean case in Appendix A. We will only use this lemma in the case when $G$ is reductive, $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ , $H$ is nilpotent and $\unicode[STIX]{x1D70C}$ is one-dimensional.

2.4 Oscillator representations of the Heisenberg group

Definition 2.4.1. Let $W_{n}$ denote the $2n$ -dimensional $\mathbb{F}$ -vector space $(\mathbb{F}^{n})^{\ast }\oplus \mathbb{F}^{n}$ and let $\unicode[STIX]{x1D714}$ be the standard symplectic form on $W_{n}$ . The Heisenberg group $H_{n}$ is the algebraic group with underlying algebraic variety $W_{n}\times \mathbb{F}$ with the group law given by

$$\begin{eqnarray}(w_{1},z_{1})(w_{2},z_{2})=(w_{1}+w_{2},z_{1}+z_{2}+1/2\unicode[STIX]{x1D714}(w_{1},w_{2})).\end{eqnarray}$$

Note that $H_{0}=\mathbb{F}$ .

Proof. In the archimedean case we apply the characterization of smooth vectors in a unitary induction given in [Reference PoulsenPou72, Theorem 5.1]. By this characterization $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D712}}$ can be identified with the space

$$\begin{eqnarray}\{f\in C^{\infty }((\mathbb{F}^{n})^{\ast })\,|\,x^{i}f^{(j)}\in L^{2}((\mathbb{F}^{n})^{\ast })\,\forall i,j\}.\end{eqnarray}$$

This space coincides with the Schwartz space ${\mathcal{S}}((\mathbb{F}^{n})^{\ast })$ , which in turn can be identified with $\text{ind}_{0\oplus \mathbb{F}^{n}\oplus \mathbb{F}}^{H_{n}}(\unicode[STIX]{x1D712})$ .

In the non-archimedean case let us prove a stronger statement: $\text{ind}_{0\oplus \mathbb{F}^{n}\oplus \mathbb{F}}^{H_{n}}(\unicode[STIX]{x1D712})=\text{Ind}_{0\oplus \mathbb{F}^{n}\oplus \mathbb{F}}^{H_{n}}(\unicode[STIX]{x1D712}),$ where $\text{Ind}$ denotes the full smooth induction. Indeed let $f\in \text{Ind}_{0\oplus \mathbb{F}^{n}\oplus \mathbb{F}}^{H_{n}}(\unicode[STIX]{x1D712}),$ and let $f^{\prime }$ be the restriction of $f$ to $(\mathbb{F}^{n})^{\ast }\oplus 0\oplus 0$ . Since $f$ is smooth, i.e. fixed by an open compact subgroup $K$ of $H_{n}$ , for any $\unicode[STIX]{x1D711}\in (\mathbb{F}^{n})^{\ast }$ and $v\in \mathbb{F}^{n}\cap K$ we have $\unicode[STIX]{x1D712}(\unicode[STIX]{x1D711}(v))f^{\prime }(\unicode[STIX]{x1D711})=f^{\prime }(\unicode[STIX]{x1D711})$ . This implies that $f^{\prime }$ has compact support, and thus $f\in \text{ind}_{0\oplus \mathbb{F}^{n}\oplus \mathbb{F}}^{H_{n}}(\unicode[STIX]{x1D712})$ .◻

2.5 Degenerate Whittaker models

Definition 2.5.1. (i) A Whittaker pair is an ordered pair $(S,\unicode[STIX]{x1D711})$ such that $S\in \mathfrak{g}$ is rational semi-simple, and $\unicode[STIX]{x1D711}\in (\mathfrak{g}^{\ast })_{-2}^{S}$ . Given such a Whittaker pair, we define the space of degenerate Whittaker models ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ in the following way: let $\mathfrak{u}:=\mathfrak{g}_{{\geqslant}1}^{S}$ . Define an anti-symmetric form $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}}$ on $\mathfrak{g}$ by $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}}(X,Y):=\unicode[STIX]{x1D711}([X,Y])$ . Let $\mathfrak{n}$ be the radical of $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}}|_{\mathfrak{u}}$ . Note that $\mathfrak{u},\mathfrak{n}$ are nilpotent subalgebras of $\mathfrak{g}$ , and $[\mathfrak{u},\mathfrak{u}]\subset \mathfrak{g}_{{\geqslant}2}^{S}\subset \mathfrak{n}$ . Let $U:=\text{Exp}(\mathfrak{u})$ and $N:=\text{Exp}(\mathfrak{n})$ be the corresponding nilpotent subgroups of $G$ . Let $\mathfrak{n}^{\prime }:=\mathfrak{n}\cap \text{Ker}(\unicode[STIX]{x1D711}),\,N^{\prime }:=\text{Exp}(\mathfrak{n}^{\prime })$ . If $\unicode[STIX]{x1D711}=0$ we define

(8) $$\begin{eqnarray}{\mathcal{W}}_{S,0}:=\text{ind}_{U}^{G}(\mathbb{C}).\end{eqnarray}$$

Assume now that $\unicode[STIX]{x1D711}$ is non-zero. Then $U/N^{\prime }$ has a natural structure of a Heisenberg group, and its center is $N/N^{\prime }$ . Let $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ denote the unitary character of $N/N^{\prime }$ given by $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}(\exp (X)):=\unicode[STIX]{x1D712}(\unicode[STIX]{x1D711}(X))$ . Let $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}}$ denote the oscillator representation of $U/N^{\prime }$ with central character $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ , and $\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}}^{\prime }$ denote its trivial lifting to $U$ . Define

(9) $$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}:=\text{ind}_{U}^{G}(\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}}^{\prime }).\end{eqnarray}$$

(ii) For a nilpotent element $\unicode[STIX]{x1D711}\in \mathfrak{g}^{\ast }$ , define the generalized Whittaker model ${\mathcal{W}}_{\unicode[STIX]{x1D711}}$ corresponding to $\unicode[STIX]{x1D711}$ to be ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ , where $S$ is a neutral element for $\unicode[STIX]{x1D711}$ if $\unicode[STIX]{x1D711}\neq 0$ and $S=0$ if $\unicode[STIX]{x1D711}=0$ . We will also call ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ neutral degenerate Whittaker model. By Lemma 2.2.1  ${\mathcal{W}}_{\unicode[STIX]{x1D711}}$ depends only on the coadjoint orbit of $\unicode[STIX]{x1D711}$ , and does not depend on the choice of $S$ . Thus, we will also use the notation ${\mathcal{W}}_{{\mathcal{O}}}$ for a nilpotent coadjoint orbit ${\mathcal{O}}\subset \mathfrak{g}^{\ast }$ . In § 5 we reformulate this definition without choosing $S$ , but using the Killing form.

(iii) For $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ define the degenerate and generalized Whittaker spaces of $\unicode[STIX]{x1D70B}$ by

(10) $$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B}):=\text{Hom}_{G}({\mathcal{W}}_{S,\unicode[STIX]{x1D711}},\unicode[STIX]{x1D70B}^{\ast })\quad \text{and}\quad {\mathcal{W}}_{\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B}):=\text{Hom}_{G}({\mathcal{W}}_{\unicode[STIX]{x1D711}},\unicode[STIX]{x1D70B}^{\ast }).\end{eqnarray}$$

Note that ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\cong \text{Hom}_{G}({\mathcal{W}}_{S,\unicode[STIX]{x1D711}},\widetilde{\unicode[STIX]{x1D70B}}),$ where $\widetilde{\unicode[STIX]{x1D70B}}$ denotes the contragredient representation. In the non-archimedean case this is obvious and in the archimedean case this follows from the Dixmier–Malliavin theorem [Reference Dixmier and MalliavinDM78].

Lemma 2.5.2. Let $\mathfrak{l}\subset \mathfrak{u}$ be a maximal isotropic subalgebra and $L:=\text{Exp}(\mathfrak{l})$ . Let $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ . Then

$$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\cong \text{Hom}_{L}(\unicode[STIX]{x1D70B},\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}^{-1}).\end{eqnarray}$$

Proof. By Corollary 2.4.5 and Lemma 2.3.2 we have ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}\cong \text{ind}_{L}^{G}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})$ . Using Lemma 2.3.4 we obtain

$$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\cong \text{Hom}_{G}(\text{ind}_{L}^{G}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}),\unicode[STIX]{x1D70B}^{\ast })\cong \text{Hom}_{L}(\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}},\unicode[STIX]{x1D70B}^{\ast })\cong \text{Hom}_{L}(\unicode[STIX]{x1D70B},\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}^{-1}).\square\end{eqnarray}$$

In the case when $\mathbb{F}$ is non-archimedean and $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ , slightly different degenerate Whittaker models are considered in [Reference Moeglin and WaldspurgerMW87]. Namely, let $U^{\prime \prime }$ denote the subgroup of $U$ generated by $\text{Exp}(\mathfrak{g}_{{>}1}^{S})$ and the kernel of $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ . Let $\unicode[STIX]{x1D70B}_{(U^{\prime \prime },\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})}$ denote the biggest quotient of $\unicode[STIX]{x1D70B}$ on which $U^{\prime \prime }$ acts by the character $\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ . Then [Reference Moeglin and WaldspurgerMW87] considers $\text{Hom}_{U}(\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}},\unicode[STIX]{x1D70B}_{(U^{\prime \prime },\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})})$ . By Lemma 2.4.6 and Frobenius reciprocity we have

(11) $$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\cong (\text{Hom}_{U}(\unicode[STIX]{x1D70E}_{\unicode[STIX]{x1D711}},\unicode[STIX]{x1D70B}_{(U^{\prime \prime },\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}})}))^{\ast }.\end{eqnarray}$$

3.1 Proof of Lemma 3.0.2

We will need the following lemma.

3.2 Proof of Theorem 3.0.1

Let $\unicode[STIX]{x1D714}$ denote the anti-symmetric form $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}}$ on $\mathfrak{g}$ defined by $\unicode[STIX]{x1D714}(X,Y):=\unicode[STIX]{x1D711}([X,Y])$ .

Our proof is based on the following lemma, which is in the spirit of [Reference Ginzburg, Rallis and SoudryGRS99, Lemma 2.2], [Reference Ginzburg, Rallis and SoudryGRS11, Lemma 7.1] or [Reference Lapid and MaoLM15a, Lemma A.1].

Lemma 3.2.1. Let $\mathfrak{l},\mathfrak{r}\subset \mathfrak{g}$ be nilpotent subalgebras such that $[\mathfrak{l},\mathfrak{r}]\subset \mathfrak{l}\cap \mathfrak{r}$ , $\unicode[STIX]{x1D714}|_{\mathfrak{l}}=0$ , $\unicode[STIX]{x1D714}|_{\mathfrak{r}}=0$ and the radical of $\unicode[STIX]{x1D714}|_{\mathfrak{l}+\mathfrak{r}}$ is $\mathfrak{l}\cap \mathfrak{r}$ . Then $\mathfrak{l}+\mathfrak{r}$ is a nilpotent Lie algebra and

(12) $$\begin{eqnarray}\text{ind}_{\text{Exp}(\mathfrak{l})}^{\text{Exp}(\mathfrak{l}+\mathfrak{r})}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r})}^{\text{Exp}(\mathfrak{l}+\mathfrak{r})}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}.\end{eqnarray}$$

Remark 3.2.2. By induction by stages we obtain $\text{ind}_{\text{Exp}(\mathfrak{l})}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r})}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}.$ Observe that this isomorphism can be realized explicitly as an integral transform: given $f\in \text{ind}_{\text{Exp}(\mathfrak{l})}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ we can define $\check{f}\in \text{ind}_{\text{Exp}(\mathfrak{r})}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}$ simply by setting

$$\begin{eqnarray}\check{f}(g)=\int _{\text{Exp}(\mathfrak{l}\cap \mathfrak{r})\backslash \text{Exp}(\mathfrak{r})}f(ng)\,dn.\end{eqnarray}$$

Then the previous results imply that the map $f\mapsto \check{f}$ defines an isomorphism between these two spaces.

In the course of our proof we will make several choices and introduce some notation. The reader is welcome to track those on Examples 3.2.11 and 3.2.12 below.

Let $z:=S-h$ and $K:=\widetilde{S}-z$ . Choose a symmetric bilinear non-degenerate $G$ -invariant form on $\mathfrak{g}$ and let $f\in \mathfrak{g}$ correspond to $\unicode[STIX]{x1D711}$ using this form. Let $e$ be the nil-positive element for $h$ and $f$ . Then $e\in \mathfrak{a}:=\mathfrak{g}_{z}$ . Consider the embedding $\mathfrak{a}^{\ast }{\hookrightarrow}\mathfrak{g}^{\ast }$ corresponding to the bilinear form on $\mathfrak{g}$ . Let $A:=G_{z}$ .

Proof. Note that $\mathfrak{a}^{\ast }=(\mathfrak{a}^{\ast })^{e}\oplus \text{ad}^{\ast }(f)(\mathfrak{a}^{\ast })$ . Since $K$ preserves both summands we get

(13) $$\begin{eqnarray}(\mathfrak{a}^{\ast })_{-2}^{K}=((\mathfrak{a}^{\ast })^{e})_{-2}^{K}\oplus (\text{ad}^{\ast }(f)(\mathfrak{a}))_{-2}^{K}=((\mathfrak{a}^{\ast })^{e})_{-2}^{K}\oplus \text{ad}^{\ast }(\mathfrak{a}_{K})(\unicode[STIX]{x1D711}).\end{eqnarray}$$

Consider the map $\unicode[STIX]{x1D708}:A_{K}\times ((\mathfrak{a}^{\ast })^{e})_{-2}^{K}\rightarrow (\mathfrak{a}^{\ast })_{-2}^{K}$ given by $\unicode[STIX]{x1D708}(g,X):=g(\unicode[STIX]{x1D711}+X)$ . Note that the differential of $\unicode[STIX]{x1D708}$ at the point $(1,0)$ is onto, and thus the image of $\unicode[STIX]{x1D708}$ contains an open neighborhood of $\unicode[STIX]{x1D708}(1,0)=\unicode[STIX]{x1D711}$ . Since $\unicode[STIX]{x1D711}\in \overline{G_{\widetilde{S}}\tilde{\unicode[STIX]{x1D711}}}$ , the image of $\unicode[STIX]{x1D708}$ intersects the orbit $G_{\widetilde{S}}\tilde{\unicode[STIX]{x1D711}}$ .◻

Let $\tilde{\unicode[STIX]{x1D711}}^{\prime }:=\unicode[STIX]{x1D711}+\unicode[STIX]{x1D711}^{\prime }$ . Let $i$ denote the smallest of the $h$ -weights of $\unicode[STIX]{x1D711}^{\prime }$ . If $\unicode[STIX]{x1D711}^{\prime }=0$ we take $i$ to be 0. Note that $i$ is always non-negative.

Let $Z:=\widetilde{S}-S=K-h\in \mathfrak{g}_{\unicode[STIX]{x1D711}}$ . For any rational number $0\leqslant t\leqslant 1$ define

(14) $$\begin{eqnarray}S_{t}:=S+tZ,\quad \mathfrak{u}_{t}:=\mathfrak{g}_{{\geqslant}1}^{S_{t}},\quad \mathfrak{v}_{t}:=\mathfrak{g}_{{>}1}^{S_{t}},\quad \text{and}\quad \mathfrak{w}_{t}:=\mathfrak{g}_{1}^{S_{t}}.\end{eqnarray}$$

Note that there are only finitely many critical numbers.

Choose a Lagrangian $\mathfrak{m}\subset \mathfrak{g}_{0}^{Z}\cap \mathfrak{g}_{1}^{S}$ and let

(15) $$\begin{eqnarray}\mathfrak{l}_{t}:=\mathfrak{m}+(\mathfrak{u}_{t}\cap \mathfrak{g}_{{<}0}^{Z})+\text{Ker}(\unicode[STIX]{x1D714}|_{\mathfrak{ u}_{t}})\quad \text{and}\quad \mathfrak{r}_{t}:=\mathfrak{m}+(\mathfrak{u}_{t}\cap \mathfrak{g}_{{>}0}^{Z})+\text{Ker}(\unicode[STIX]{x1D714}|_{\mathfrak{ u}_{t}}).\end{eqnarray}$$

Define

(16) $$\begin{eqnarray}\mathfrak{l}_{t}^{\prime }:=\mathfrak{l}_{t}\cap \text{Ker}(\unicode[STIX]{x1D711}^{\prime })\quad \text{and}\quad \mathfrak{r}_{t}^{\prime }:=\mathfrak{r}_{t}\cap \text{Ker}(\unicode[STIX]{x1D711}^{\prime }).\end{eqnarray}$$

Lemmas 3.2.8 and 3.2.1 imply that

(17) $$\begin{eqnarray}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r}_{0})}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r}_{0}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}},\quad \text{ind}_{\text{Exp}(\mathfrak{l}_{t}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r}_{t}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\end{eqnarray}$$

and for $s<t$ such that all the numbers in $(s,t)$ are regular we have

(18) $$\begin{eqnarray}\text{ind}_{\text{Exp}(\mathfrak{r}_{s}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\,{\twoheadrightarrow}\,\text{ind}_{\text{Exp}(\mathfrak{l}_{t}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}.\end{eqnarray}$$

We are now ready to prove Theorem 3.0.1.

Proof of Theorem 3.0.1.

Let $0<t_{1}<\cdots <t_{n}$ be the critical numbers. Note that $\mathfrak{r}_{t_{n}}^{\prime }$ is an isotropic subalgebra of $\mathfrak{u}_{1}$ , and that it is also isotropic with respect to the form $\unicode[STIX]{x1D714}_{\unicode[STIX]{x1D711}+\unicode[STIX]{x1D711}^{\prime }}(X,Y):=(\unicode[STIX]{x1D711}+\unicode[STIX]{x1D711}^{\prime })([X,Y])$ . Let $\mathfrak{q}$ be a maximal isotropic subspace of $\mathfrak{u}_{1}$ with respect to this form. It includes $\mathfrak{v}_{1}$ and thus is necessary a subalgebra. Note that

(19) $$\begin{eqnarray}\text{ind}_{\text{Exp}(\mathfrak{r}_{t_{n}}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}=\text{ind}_{\text{Exp}(\mathfrak{r}_{t_{n}}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}+\unicode[STIX]{x1D711}^{\prime }}\,{\twoheadrightarrow}\,\text{ind}_{\text{Exp}(\mathfrak{q})}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}+\unicode[STIX]{x1D711}^{\prime }}\simeq {\mathcal{W}}_{\widetilde{S},\tilde{\unicode[STIX]{x1D711}}^{\prime }}\simeq {\mathcal{W}}_{\widetilde{S},\tilde{\unicode[STIX]{x1D711}}}.\end{eqnarray}$$

Thus,

(20) $$\begin{eqnarray}\displaystyle {\mathcal{W}}_{S,\unicode[STIX]{x1D711}} & \simeq & \displaystyle \text{ind}_{\text{Exp}(\mathfrak{r}_{0}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\,{\twoheadrightarrow}\,\text{ind}_{\text{Exp}(\mathfrak{l}_{t_{1}}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r}_{t_{1}}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}{\twoheadrightarrow}\nonumber\\ \displaystyle & & \displaystyle \cdots \,{\twoheadrightarrow}\,\text{ind}_{\text{Exp}(\mathfrak{l}_{t_{n}}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\simeq \text{ind}_{\text{Exp}(\mathfrak{r}_{t_{n}}^{\prime })}^{G}\,\unicode[STIX]{x1D712}_{\unicode[STIX]{x1D711}}\,{\twoheadrightarrow}\,{\mathcal{W}}_{\widetilde{S},\tilde{\unicode[STIX]{x1D711}}}.\end{eqnarray}$$

◻

Let us now present two examples for the elements and subalgebras defined in the course of the proof. Let $G:=\text{GL}(4,\mathbb{F})$ and define $\unicode[STIX]{x1D711}$ by $\unicode[STIX]{x1D711}(X):=\text{Tr}(X(E_{21}+E_{43}))$ , where $E_{21}$ and $E_{43}$ are elementary matrices. Let $h$ be the diagonal matrix $\text{diag}(1,-1,1,-1)$ and $S:=h$ .

Example 3.2.11. Let $\tilde{\unicode[STIX]{x1D711}}:=\unicode[STIX]{x1D711}$ , $K:=\text{diag}(3,1,-1,-3)$ , $z:=0$ . Then $Z=\text{diag}(2,2,-2,-2),S_{t}=\text{diag}(1+2t,-1+2t,1-2t,-1-2t)$ and the weights of $S_{t}$ are as follows:

$$\begin{eqnarray}\left(\begin{array}{@{}cccc@{}}0 & 2 & 4t & 4t+2\\ -2 & 0 & 4t-2 & 4t\\ -4t & -4t+2 & 0 & 2\\ -4t-2 & -4t & -2 & 0\end{array}\right).\end{eqnarray}$$

The critical numbers are $1/4$ and $3/4$ . For $t\geqslant 3/4$ we get the principal degenerate Whittaker model. We have $\mathfrak{r}_{t}^{\prime }=\mathfrak{r}_{t},\mathfrak{l}_{t}^{\prime }=\mathfrak{l}_{t}$ for all $t$ . The above system of inclusions of $\mathfrak{r}_{0}\subset \mathfrak{l}_{1/4}\sim \mathfrak{r}_{1/4}\subset \mathfrak{l}_{3/4}=\mathfrak{r}_{3/4}$ is

$$\begin{eqnarray}\left(\begin{array}{@{}cccc@{}}0 & - & 0 & -\\ 0 & 0 & 0 & 0\\ 0 & - & 0 & -\\ 0 & 0 & 0 & 0\end{array}\right)\subset \left(\begin{array}{@{}cccc@{}}0 & - & a & -\\ 0 & 0 & 0 & a\\ 0 & \ast & 0 & -\\ 0 & 0 & 0 & 0\end{array}\right)\sim \left(\begin{array}{@{}cccc@{}}0 & - & \ast & -\\ 0 & 0 & 0 & \ast \\ 0 & 0 & 0 & -\\ 0 & 0 & 0 & 0\end{array}\right)\subset \left(\begin{array}{@{}cccc@{}}0 & - & - & -\\ 0 & 0 & \ast & -\\ 0 & 0 & 0 & -\\ 0 & 0 & 0 & 0\end{array}\right).\end{eqnarray}$$

Here, both $\ast$ and $-$ denote arbitrary elements: $-$ denotes the entries in $\mathfrak{v}_{t}$ and $\ast$ those in $\mathfrak{w}_{t}$ . The letter $a$ denotes an arbitrary element, but the two appearances of $a$ denote the same numbers. The passage from $\mathfrak{l}_{1/4}$ to $\mathfrak{r}_{1/4}$ is denoted by ${\sim}$ . At $3/4$ we have $\mathfrak{l}_{3/4}=\mathfrak{r}_{3/4}$ .

Let us now give an example in which $\unicode[STIX]{x1D711}$ and $\tilde{\unicode[STIX]{x1D711}}$ are not equal and not conjugate.

Example 3.2.12. Identify $\mathfrak{g}\simeq \mathfrak{g}^{\ast }$ using the trace form and let

$$\begin{eqnarray}\tilde{\unicode[STIX]{x1D711}}=\left(\begin{array}{@{}cccc@{}}0 & 0 & 1 & 0\\ 1 & 0 & 0 & 1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 1 & 0\end{array}\right),\quad e=\left(\begin{array}{@{}cccc@{}}0 & 1 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0\end{array}\right),\quad \unicode[STIX]{x1D711}^{\prime }=\left(\begin{array}{@{}cccc@{}}0 & 0 & 1 & 0\\ 0 & 0 & 0 & 1\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\end{array}\right).\end{eqnarray}$$

Let $\widetilde{S}:=\text{diag}(0,-2,2,0)$ , $Z=\text{diag}(-1,-1,1,1)$ . Note that $\mathfrak{w}_{0}=\mathfrak{w}_{1}=0$ and the only critical value of $t$ is $1/2$ . The sequence of subalgebras from the proof of Theorem 3.0.1 is

$$\begin{eqnarray}\mathfrak{u}_{0}=l_{0}^{\prime }=\mathfrak{l}_{1/2}^{\prime }=\left(\begin{array}{@{}cccc@{}}0 & \ast & 0 & -\\ 0 & 0 & 0 & 0\\ 0 & \ast & 0 & \ast \\ 0 & 0 & 0 & 0\end{array}\right)\sim \mathfrak{r}_{1/2}^{\prime }=\left(\begin{array}{@{}cccc@{}}0 & \ast & 0 & 0\\ 0 & 0 & 0 & 0\\ a & \ast & 0 & \ast \\ 0 & -a & 0 & 0\end{array}\right)\subset \left(\begin{array}{@{}cccc@{}}0 & \ast & 0 & 0\\ 0 & 0 & 0 & 0\\ \ast & \ast & 0 & \ast \\ 0 & \ast & 0 & 0\end{array}\right)=\mathfrak{u}_{1}.\end{eqnarray}$$

3.3 Principal degenerate Whittaker models and proof of theorem B

In the discussion below $\mathfrak{a}$ will denote a maximal split toral subalgebra of $\mathfrak{g}$ , we will write $\unicode[STIX]{x1D6F4}(\mathfrak{a,g})$ for the corresponding (restricted) root system, $\unicode[STIX]{x1D6F4}^{+}(\mathfrak{a,g})$ for a choice of positive roots, and $\unicode[STIX]{x1D6E5}(\mathfrak{a,g})$ for the corresponding system of simple roots.

We fix a non-degenerate invariant bilinear form on $\mathfrak{g}$ , which allows us to identify nilpotent elements in $\mathfrak{g}$ and $\mathfrak{g}^{\ast }$ . Thus, we may equally well apply the above terminology to ‘Whittaker pairs’ $(S,f)\subset \mathfrak{g}\times \mathfrak{g}$ , such that $S$ is rational semisimple and $[S,f]=-2f$ .

Note that a principal element $S$ uniquely determines both $\mathfrak{a}$ , and the simple system $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}(\mathfrak{a,g})$ . Thus, there is a bijection between principal elements and simple systems which we denote by $S\mapsto \unicode[STIX]{x1D6E5}_{S},\text{ }\unicode[STIX]{x1D6E5}\mapsto S_{\unicode[STIX]{x1D6E5}}$ .

If $(S,f)$ is principal, then setting $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}_{S}$ , we can write $f$ uniquely in the form

(21) $$\begin{eqnarray}f=\mathop{\sum }_{\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6E5}}Y_{\unicode[STIX]{x1D6FC}},\end{eqnarray}$$

for some $Y_{\unicode[STIX]{x1D6FC}}\in \mathfrak{g}_{-\unicode[STIX]{x1D6FC}}$ . Conversely if $f$ is of the form (21) for some $\unicode[STIX]{x1D6E5}$ , then we will say that $f$ is a PL nilpotent and $\unicode[STIX]{x1D6E5}$ is compatible with $f$ . In this case $\left(S_{\unicode[STIX]{x1D6E5}},f\right)$ is a principal Whittaker pair, and we define the $\unicode[STIX]{x1D6E5}$ -support of $f$ to be

$$\begin{eqnarray}\text{supp}_{\unicode[STIX]{x1D6E5}}(f)=\{\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6E5}\mid Y_{\unicode[STIX]{x1D6FC}}\neq 0\}.\end{eqnarray}$$

If $\text{supp}_{\unicode[STIX]{x1D6E5}}\left(f\right)=\unicode[STIX]{x1D6E5}$ we say that $f$ is a principal nilpotent element. Note that this notion is weaker than the similar notion defined in [Reference BourbakiBou75, VIII.11.4]. However, if $G$ is quasi-split then both notions are equivalent to the notion of regular nilpotent element.

Note that $f$ is a PL nilpotent if and only if it is a principal nilpotent element for a Levi subgroup of $G$ . For the general linear groups, every orbit includes such an element. For complex classical groups, all such orbits are described in [Reference Gourevitch and SahiGS15, § 6] in terms of the corresponding partitions. For complex exceptional groups, these are the orbits with non-parenthetical Bala–Carter labels.

Lemma 3.3.2. Let $(S_{1},f)$ and $(S_{2},f)$ be two Whittaker pairs with the same $f$ , and let $h$ be neutral for $f$ . Then there exist $g_{1},g_{2}\in G_{\unicode[STIX]{x1D711}}$ and $Z_{1},Z_{2}\in \mathfrak{g}_{h}\cap \mathfrak{g}_{f}$ such that

$$\begin{eqnarray}g_{1}\cdot S_{1}=h+Z_{1},\quad g_{2}\cdot S_{2}=h+Z_{2},\quad [Z_{1},Z_{2}]=0.\end{eqnarray}$$

Proof. Let us first choose any simple system $\unicode[STIX]{x1D6E5}(\mathfrak{b,g})$ compatible with $f$ . By Lemma 3.3.2 there exists $g$ in the centralizer $G_{f}$ of $f$ such that $g\cdot S\in \mathfrak{b}$ . Now the action of $g^{-1}$ carries $\mathfrak{b}$ to a maximal toral subalgebra $\mathfrak{a}$ and $\unicode[STIX]{x1D6E5}(\mathfrak{b,g})$ to a simple system $\unicode[STIX]{x1D6E5}^{\prime }=\unicode[STIX]{x1D6E5}^{\prime }(\mathfrak{a,g})$ which satisfies parts (a) and (b). Next note that if $S^{\prime }=S_{\unicode[STIX]{x1D6E5}^{\prime }}$ , then $(S^{\prime },f)$ is a Whittaker pair, so by Lemma 3.0.2 there is a neutral element $h$ for $f$ that commutes with $S_{\unicode[STIX]{x1D6E5}^{\prime }}$ ; but this forces $h\in \mathfrak{a}$ and thus part (c) holds.

If $\text{supp}_{\unicode[STIX]{x1D6E5}^{\prime }}(f)=\unicode[STIX]{x1D6E5}^{\prime }$ , then we set $\unicode[STIX]{x1D6E5}=\unicode[STIX]{x1D6E5}^{\prime }$ ; in this case we have $h-S$ is central and part (d) holds vacuously. Now suppose $\text{supp}_{\unicode[STIX]{x1D6E5}^{\prime }}\left(f\right)\subsetneq \unicode[STIX]{x1D6E5}^{\prime }$ . Then we will show how to modify $\unicode[STIX]{x1D6E5}^{\prime }$ to obtain a new system $\unicode[STIX]{x1D6E5}(\mathfrak{a,g})$ such that parts (a)–(d) are satisfied. For this let us write

$$\begin{eqnarray}Z=S-h,\quad Z^{\prime }=S^{\prime }-h,\quad h_{\unicode[STIX]{x1D700}}=Z+\unicode[STIX]{x1D700}Z^{\prime }+\unicode[STIX]{x1D700}^{2}h,\end{eqnarray}$$

where $\unicode[STIX]{x1D700}>0$ is chosen sufficiently small so that for all $\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}(\mathfrak{a,g})$ we have

(22) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FC}(Z)>0 & \;\Longrightarrow \; & \displaystyle \unicode[STIX]{x1D6FC}(Z)>\unicode[STIX]{x1D700}|\unicode[STIX]{x1D6FC}(Z^{\prime })|+\unicode[STIX]{x1D700}^{2}|\unicode[STIX]{x1D6FC}(h)|,\end{eqnarray}$$

(23) $$\begin{eqnarray}\displaystyle \unicode[STIX]{x1D6FC}(Z^{\prime })>0 & \;\Longrightarrow \; & \displaystyle \unicode[STIX]{x1D6FC}(Z^{\prime })>\unicode[STIX]{x1D700}|\unicode[STIX]{x1D6FC}(h)|.\end{eqnarray}$$

We claim that $h_{\unicode[STIX]{x1D700}}$ is regular in the sense that $\unicode[STIX]{x1D6FC}(h_{\unicode[STIX]{x1D700}})\neq 0\text{ for all }\unicode[STIX]{x1D6FC}.$ If $\unicode[STIX]{x1D6FC}(Z)>0$ this follows from (22), if $\unicode[STIX]{x1D6FC}(Z)<0$ , we simply replace $\unicode[STIX]{x1D6FC}$ by $-\unicode[STIX]{x1D6FC}$ . If $\unicode[STIX]{x1D6FC}(Z)=0$ but $\unicode[STIX]{x1D6FC}(Z^{\prime })\neq 0$ , then this follows analogously from (23). Finally, if $\unicode[STIX]{x1D6FC}(Z)=\unicode[STIX]{x1D6FC}(Z^{\prime })=0$ , then $\unicode[STIX]{x1D6FC}(h_{\unicode[STIX]{x1D700}})=\unicode[STIX]{x1D700}^{2}\unicode[STIX]{x1D6FC}(h)$ and we must show that $\unicode[STIX]{x1D6FC}(h)\neq 0$ , but this follows from the regularity of $S^{\prime }=h+Z^{\prime }$ .

This means that we can define a positive root system as follows

$$\begin{eqnarray}\unicode[STIX]{x1D6F4}^{+}(\mathfrak{a,g})=\{\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}\mid \unicode[STIX]{x1D6FC}(h_{\unicode[STIX]{x1D700}})>0\}.\end{eqnarray}$$

Let $\unicode[STIX]{x1D6E5}$ be the corresponding simple system; we will show that $\text{supp}_{\unicode[STIX]{x1D6E5}^{\prime }}(f)\subset \unicode[STIX]{x1D6E5}.$ This implies that $\unicode[STIX]{x1D6E5}$ is compatible with $e$ , so that part (a) holds. To prove that $\text{supp}_{\unicode[STIX]{x1D6E5}^{\prime }}(f)\subset \unicode[STIX]{x1D6E5}$ suppose $\unicode[STIX]{x1D6FC}$ belongs to $\text{supp}_{\unicode[STIX]{x1D6E5}^{\prime }}(f)$ . Then we have $\unicode[STIX]{x1D6FC}(Z)=\unicode[STIX]{x1D6FC}(Z^{\prime })=0,\quad \unicode[STIX]{x1D6FC}(h_{\unicode[STIX]{x1D700}})=\unicode[STIX]{x1D700}^{2}\unicode[STIX]{x1D6FC}(h)=2\unicode[STIX]{x1D700}^{2}>0$ . This means that $\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}^{+}(\mathfrak{a,g})$ and it remains to show that we cannot write

(24) $$\begin{eqnarray}\unicode[STIX]{x1D6FC}=\unicode[STIX]{x1D6FD}+\unicode[STIX]{x1D6FE}\end{eqnarray}$$

where $\unicode[STIX]{x1D6FD},\unicode[STIX]{x1D6FE}\in \unicode[STIX]{x1D6F4}^{+}(\mathfrak{a,g})$ . Now if $\unicode[STIX]{x1D6FD}(Z)>0$ or $\unicode[STIX]{x1D6FD}(Z^{\prime })>0$ , then (24) is impossible by (22) and (23). Thus, we may assume $\unicode[STIX]{x1D6FD}(Z)=\unicode[STIX]{x1D6FD}(Z^{\prime })=0$ and thus $\unicode[STIX]{x1D6FD}(S^{\prime })=\unicode[STIX]{x1D6FD}(Z^{\prime })+\unicode[STIX]{x1D6FD}(h)=2>0,$ and similarly $\unicode[STIX]{x1D6FE}(S^{\prime })>0$ . But $\unicode[STIX]{x1D6FC}$ is a simple root in the positive system defined by $S^{\prime }$ , so (24) cannot hold.

Finally we verify that $\unicode[STIX]{x1D6F4}^{+}(\mathfrak{a,g})$ satisfies part (d). Thus suppose $\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}(\mathfrak{a,g})$ satisfies $\unicode[STIX]{x1D6FC}(h)\leqslant 0$ and $\unicode[STIX]{x1D6FC}(S)>0$ , then we have $\unicode[STIX]{x1D6FC}(Z)=\unicode[STIX]{x1D6FC}(S)-\unicode[STIX]{x1D6FC}(h)>0.$ By (22), this implies $\unicode[STIX]{x1D6FC}(h_{\unicode[STIX]{x1D700}})=\unicode[STIX]{x1D6FC}(Z)+\unicode[STIX]{x1D700}\unicode[STIX]{x1D6FC}(Z^{\prime })+\unicode[STIX]{x1D700}^{2}\unicode[STIX]{x1D6FC}(h)>0$ . Thus, we have $\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}^{+}(\mathfrak{a,g})$ as desired.◻

Proof. By Corollary 3.0.5 it suffices to find a maximally split toral subalgebra $\mathfrak{a}$ and containing a neutral element $h$ for $\unicode[STIX]{x1D711}$ , and a principal element $\tilde{S}$ such that $\tilde{S}-h\in \mathfrak{a}_{\unicode[STIX]{x1D711}}$ and $S-h\geqslant _{\unicode[STIX]{x1D711}}\tilde{S}-h$ in the sense of Definition 3.0.4, i.e.

$$\begin{eqnarray}\unicode[STIX]{x1D6FC}(h)\leqslant 0\quad \text{and}\quad \unicode[STIX]{x1D6FC}(S)\geqslant 1\;\Longrightarrow \;\unicode[STIX]{x1D6FC}(\tilde{S})\geqslant 1\quad \text{for all }\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}(\mathfrak{a,g}).\end{eqnarray}$$

Let $f\in \mathfrak{g}$ be the nilpotent element corresponding to $\unicode[STIX]{x1D711}$ . Then $(S,f)$ satisfies the conditions of Proposition 3.3.3. Let $\mathfrak{a},h,\unicode[STIX]{x1D6F4}^{+},\unicode[STIX]{x1D6E5}$ be as in the proposition, and let $\tilde{S}=S_{\unicode[STIX]{x1D6E5}}$ . Then any $\unicode[STIX]{x1D6FC}\in \unicode[STIX]{x1D6F4}(\mathfrak{a,g})$ with $\unicode[STIX]{x1D6FC}(h)\leqslant 0\text{ and }\unicode[STIX]{x1D6FC}(S)\geqslant 1$ lies in $\unicode[STIX]{x1D6F4}^{+}$ . Since $\tilde{S}$ is principal, we deduce that $\unicode[STIX]{x1D6FC}(\tilde{S})\geqslant 2$ .◻

3.3.1 Archimedean case

For a smooth representation $\unicode[STIX]{x1D70B}$ of a real reductive group $G$ one can define one more invariant, which we denote ${\mathcal{V}}(\unicode[STIX]{x1D70B})$ and call the annihilator variety of $\unicode[STIX]{x1D70B}$ . It is sometimes called the associated variety of the annihilator of $\unicode[STIX]{x1D70B}$ . It is defined to be the set of zeros in $\mathfrak{g}_{\mathbb{C}}^{\ast }$ of the ideal in the symmetric algebra $S(\mathfrak{g}_{\mathbb{C}})$ , which is generated by the symbols of the annihilator ideal of $\unicode[STIX]{x1D70B}$ in the universal enveloping algebra $U(\mathfrak{g}_{\mathbb{C}})$ . It follows from [Reference VoganVog91, Theorem 8.4] and [Reference Schmid and VilonenSV00] that ${\mathcal{V}}(\unicode[STIX]{x1D70B})$ is the Zariski closure of $\operatorname{WF}(\unicode[STIX]{x1D70B})$ in $\mathfrak{g}_{\mathbb{C}}^{\ast }$ . Note that if $G$ is a complex reductive group or $G=\text{GL}_{n}(\mathbb{R})$ we have $\operatorname{WF}(\unicode[STIX]{x1D70B})={\mathcal{V}}(\unicode[STIX]{x1D70B})\cap \mathfrak{g}^{\ast }$ .

We will use the following theorems.

Theorem 3.3.5 [Reference MatumotoMat87, Corollary 4].

Let $\unicode[STIX]{x1D70B}$ be a smooth representation of $G$ , let ${\mathcal{O}}\subset \mathfrak{g}^{\ast }$ be a nilpotent orbit and suppose that ${\mathcal{W}}_{{\mathcal{O}}}(\unicode[STIX]{x1D70B})\neq 0$ . Then ${\mathcal{O}}\subset {\mathcal{V}}(\unicode[STIX]{x1D70B})$ .

Theorem 3.3.6 [Reference Gourevitch and SahiGS15, Theorem B].

Let $(S,\unicode[STIX]{x1D711})$ be a principal degenerate Whittaker pair for $G$ . Let $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ such that $\unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B})$ . Then:

1. (i) if $G$ is a complex group or $G=\text{GL}_{n}(\mathbb{R})$ , then ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\neq 0$ ;

2. (ii) if $G$  is quasisplit, then there exists $g\in G_{\mathbb{C}}$ such that $\text{ad}(g)$ preserves $\mathfrak{g}$ and ${\mathcal{W}}_{\text{ad}(g)(S),\text{ad}(g)(\unicode[STIX]{x1D711})}(\unicode[STIX]{x1D70B})\neq 0$ .

In fact, the theorems in [Reference MatumotoMat87, Reference Gourevitch and SahiGS15] are stronger than the versions we state here.

Proof of Theorem B.

Let $(S,\unicode[STIX]{x1D711})$ be as in the theorem. Then $\unicode[STIX]{x1D711}$ can be completed to a principal Whittaker pair, and Proposition 3.3.4 implies that the degenerate Whittaker model ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ can be mapped onto some principal degenerate Whittaker model ${\mathcal{W}}_{\widetilde{S},\unicode[STIX]{x1D711}}$ . By Theorem A, ${\mathcal{W}}_{\unicode[STIX]{x1D711}}$ maps onto ${\mathcal{W}}_{S,\unicode[STIX]{x1D711}}$ . Thus, we have ${\mathcal{W}}_{\widetilde{S},\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B}){\hookrightarrow}{\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B}){\hookrightarrow}{\mathcal{W}}_{\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})$ . Together with Theorems 3.3.5 and 3.3.6 we get

$$\begin{eqnarray}\unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B})\Rightarrow {\mathcal{W}}_{\widetilde{S},\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\neq 0\Rightarrow {\mathcal{W}}_{S,\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\neq 0\Rightarrow {\mathcal{W}}_{\unicode[STIX]{x1D711}}(\unicode[STIX]{x1D70B})\neq 0\Rightarrow \unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B}).\square\end{eqnarray}$$

In the same way we get the following statement for real reductive groups.

Corollary 3.3.7. Suppose that $G$  is quasisplit and let $\unicode[STIX]{x1D70B}\in {\mathcal{M}}(G)$ . Let $(S,\unicode[STIX]{x1D711})$ be a Whittaker pair such that $\unicode[STIX]{x1D711}$ is a PL nilpotent. Suppose that $\unicode[STIX]{x1D711}\in \operatorname{WF}(\unicode[STIX]{x1D70B})$ . Then there exists $g\in G_{\mathbb{C}}$ such that $\text{ad}(g)$ preserves $\mathfrak{g}$ and ${\mathcal{W}}_{\text{ad}(g)(S),\text{ad}(g)(\unicode[STIX]{x1D711})}(\unicode[STIX]{x1D70B})\neq 0$ .

4.1 Notation

Let us first introduce some notation. A composition $\unicode[STIX]{x1D702}$ of $n$ is a sequence of natural (positive) numbers $\unicode[STIX]{x1D702}_{1},\ldots ,\unicode[STIX]{x1D702}_{k}$ with $\sum \unicode[STIX]{x1D702}_{i}=n$ . The length of $\unicode[STIX]{x1D702}$ is $k$ . A partition $\unicode[STIX]{x1D706}$ is a composition such that $\unicode[STIX]{x1D706}_{1}\geqslant \unicode[STIX]{x1D706}_{2}\geqslant \cdots \geqslant \unicode[STIX]{x1D706}_{k}$ . For a composition $\unicode[STIX]{x1D702}$ we denote by $\unicode[STIX]{x1D702}^{{\geqslant}}$ the corresponding partition. A partial order on partitions of $n$ is defined by

(25) $$\begin{eqnarray}\unicode[STIX]{x1D706}\geqslant \unicode[STIX]{x1D707}\quad \text{if }\mathop{\sum }_{i=1}^{j}\unicode[STIX]{x1D706}_{i}\geqslant \mathop{\sum }_{i=1}^{j}\unicode[STIX]{x1D707}_{i}\quad \text{for any }1\leqslant j\leqslant \text{length}(\unicode[STIX]{x1D706}),\text{length}(\unicode[STIX]{x1D707}).\end{eqnarray}$$

We will use the notation $\text{diag}(x_{1},\ldots ,x_{k})$ for diagonal and block-diagonal matrices. For a natural number $k$ we denote by $J_{k}\in \mathfrak{g}_{k}$ the lower-triangular Jordan block of size $k$ , and by $h_{k}$ the diagonal matrix $h_{k}:=\text{diag}(k-1,k-3,\ldots ,1-k)$ . For a composition $\unicode[STIX]{x1D702}$ we denote

(26) $$\begin{eqnarray}J_{\unicode[STIX]{x1D702}}:=\text{diag}(J_{\unicode[STIX]{x1D702}_{1}},\ldots ,J_{\unicode[STIX]{x1D702}_{k}})\in \mathfrak{g}_{n}\quad \text{and}\quad h_{\unicode[STIX]{x1D702}}:=\text{diag}(h_{\unicode[STIX]{x1D702}_{1}},\ldots ,h_{\unicode[STIX]{x1D702}_{k}})\in \mathfrak{g}_{n}.\end{eqnarray}$$

Note that $[h_{\unicode[STIX]{x1D702}},J_{\unicode[STIX]{x1D702}}]=-2J_{\unicode[STIX]{x1D702}}$ and $(J_{\unicode[STIX]{x1D702}},h_{\unicode[STIX]{x1D702}})$ can be completed to an $\mathfrak{sl}_{2}$ -triple.

Identify $\mathfrak{g}_{n}^{\ast }$ with $\mathfrak{g}_{n}$ using the trace form. Denote by ${\mathcal{O}}_{\unicode[STIX]{x1D702}}$ the orbit of $J_{\unicode[STIX]{x1D702}}$ , and also the corresponding orbit in $\mathfrak{g}_{n}^{\ast }$ . By the Jordan theorem all nilpotent orbits are of this form. It is well known that ${\mathcal{O}}_{\unicode[STIX]{x1D702}}\subset \overline{{\mathcal{O}}_{\unicode[STIX]{x1D6FE}}}$ if and only if $\unicode[STIX]{x1D702}^{{\geqslant}}\leqslant \unicode[STIX]{x1D6FE}^{{\geqslant}}$ . Let $T_{n}\subset G_{n}$ denote the subgroup of diagonal matrices and $\mathfrak{t}_{n}\subset \mathfrak{g}_{n}$ the subalgebra of diagonal matrices.

4.2 Proof of Theorem D

Let $E_{ij}$ denote the elementary matrix with 1 in the $(i,j)$ entry and zeros elsewhere.

Proof. If $r=0$ or $q=0$ , we take $X:=J_{q+r}$ , $S:=h_{(p+q,r)}$ and note that $\text{diag}(J_{p},X)\in \overline{T_{m}J_{(p+q,r)}}\subset \overline{(G_{m})_{S}J_{(p+q,r)}}$ . Assume now $q,r>0$ and let

$$\begin{eqnarray}\displaystyle & F:=J_{(p+q,r)},\quad S:=\text{diag}(h_{p+q},h_{r}+(r+q-p)\text{Id}_{r})\in \mathfrak{g}_{m}, & \displaystyle \nonumber\\ \displaystyle & g:=(\text{Id}_{m}+E_{p-r+1,m-r+1})(\text{Id}_{m}+E_{p-r+2,m-r+2})\cdot \ldots \cdot (\text{Id}_{m}+E_{p,m})\in G_{m}. & \displaystyle \nonumber\end{eqnarray}$$

Note that $S_{i}:=S_{ii}=p+q-(2i-1)$ for $1\leqslant i\leqslant p+q$ and $S_{i}=2r+3q+p-(2i-1)$ for $p+q+1\leqslant i\leqslant m$ . Thus, $S_{m-r+j}=S_{p-r+j}$ for $1\leqslant j\leqslant r$ and thus $g$ commutes with $S$ . Note also that $F^{\prime }:=\text{Ad}(g)(F)=F+E_{p+1,m}$ . Conjugating $F^{\prime }$ by a suitable diagonal matrix we can obtain $F^{\prime }-(1-t)E_{p+1,p}\in (G_{m})_{S}F$ for any $t\in \mathbb{F}^{\times }$ . Letting $t$ go to zero, we get that

$$\begin{eqnarray}f:=F^{\prime }-E_{p+1,p}=F+E_{p+1,m}-E_{p+1,p}\in \overline{(G_{m})_{S}F}.\end{eqnarray}$$

Finally, it is easy to see that $f=\text{diag}(J_{p},X)$ for a regular nilpotent $X\in \mathfrak{g}_{q+r}$ .◻

We are now ready to prove Theorem D. Let us reformulate it in terms of partitions.