2.1 Notation

Throughout the paper, F will denote a non-Archimedean local field of residue characteristic q and uniformizer $\varpi $ , equipped with the absolute value $|\cdot |$ normalized in the usual way.

We let G denote the special odd orthogonal group, the symplectic group, or the (full) even orthogonal group. If we want to emphasize the rank, we use $G_n$ to denote $\mathrm {SO}(2n+1,F)$ , $\mathrm {Sp}(2n,F)$ , or $\mathrm {O}(2n,F)$ . By $\text {Rep}(G)$ , we denote the category of smooth complex representations of G.

For an arbitrary group H, we let $X(H)$ denote the group of complex characters of H.

2.2 Parabolic subgroups

If G is the disconnected group $\mathrm {O}(2n,F)$ , then, following [Reference Goldberg and Herb15], we consider only parabolic subgroups $P=MN$ such that M has supercuspidal (modulo center) representations. Explicitly, this means that

$$\begin{align*}M=\mathrm{GL}_{n_1}(F) \times \cdots \times \mathrm{GL}_{n_k} (F) \times \mathrm{O}(2n_0,F); \end{align*}$$

however, we do not allow $n_0=1$ if $\mathrm {O}(2n,F)$ is split.

2.3 Unramified characters

If M is a Levi subgroup of G, we let $M^{\circ } = \bigcap _{\chi } \ker |\chi |$ , the intersection taken over the set of all rational characters $\chi : M \to F^{\times }$ . We say that a (complex) character $\chi $ of M is unramified if it is trivial on $M^{\circ }$ ; we let $X^{\text {nr}}(M)$ denote the group of all unramified characters on M. Then $M/M^{\circ }$ is a free $\mathbb {Z}$ -module of finite rank, and the group $X^{\text {nr}}(M) = X(M/M^{\circ })$ has a natural structure of a complex affine variety. For any element $m\in M$ , we denote by $b_m$ the evaluation $\chi \mapsto \chi (m)$ .

Now, let $\sigma $ be an irreducible cuspidal representation of M, and set $M^{\sigma } = \{m \in M : {}^m{\sigma } \cong \sigma \}$ . Then $M/M^{\sigma }$ is a finite abelian group and, abusing notation, we let $\mathcal {A}$ denote the ring of regular functions on the quotient variety $X(M/M^{\circ })/X(M/M^{\sigma })$ . Since $M^{\sigma }/M^{\circ }$ is once again a free $\mathbb {Z}$ -module (of the same rank as $M/M^{\circ }$ ), we have $\mathcal {A} \cong \mathbb {C}[M^{\sigma }/M^{\circ }]$ , by $m \mapsto b_m$ . Furthermore, letting $\sigma _0$ denote an arbitrary irreducible constituent of $\sigma |_{M^{\circ }}$ , we have a canonical isomorphism $\mathcal {A} \cong {\mathrm {End}}_M({\mathrm {c-ind}}_{M^{\circ }}^M \sigma _0)$ . Indeed, this follows from a simple application of Mackey theory. We refer the reader to [Reference Heiermann17, Sections 1.17 and 4] for additional details.

2.4 The Hecke algebra of a Bernstein component

If $\pi $ is an irreducible representation of G, there is a Levi subgroup M of G and an irreducible cuspidal representation $\sigma $ of M such that $\pi $ is (isomorphic to) a subquotient of $i_P^G(\sigma )$ . Here, P is a parabolic subgroup of G with a Levi component M. The pair $(M,\sigma )$ is determined by $\pi $ up to conjugacy; we call $(M,\sigma )$ the cuspidal support of $\pi $ .

We say that the two pairs $(M_1,\sigma _1)$ and $(M_2,\sigma _2)$ as above are inertially equivalent if there exist an element $g\in G$ and an unramified character $\chi $ of $M_2$ such that

$$\begin{align*}{}^g\sigma_1 = \sigma_2 \otimes \chi. \end{align*}$$

This is an equivalence relation on the set of all pairs $(M,\sigma )$ . Given an equivalence class $[(M,\sigma )]$ , we denote by $\text {Rep}_{(M,\sigma )}(G)$ the full subcategory of $\text {Rep}(G)$ defined by the requirement that all irreducible subquotients of every object in $\text {Rep}_{(M,\sigma )}(G)$ be supported within the inertial class $[(M,\sigma )]$ . A classic result of Bernstein then shows that the category $\text {Rep}(G)$ decomposes as a direct product

$$\begin{align*}\text{Rep}(G) = \prod_{[(M,\sigma)]} \text{Rep}_{(M,\sigma)}(G) \end{align*}$$

taken over the set of all inertial equivalence classes. We refer to $\text {Rep}_{(M,\sigma )}(G)$ as the Bernstein component attached to the pair $(M,\sigma )$ . For a detailed discussion of the above results, see [Reference Bernstein3] or [Reference Bernstein4].

For each Bernstein component $\text {Rep}_{(M,\sigma )}(G)$ , one can construct a projective generator $\Gamma _{(M,\sigma )}$ by setting

$$\begin{align*}\Gamma_{(M,\sigma)} = i_P^G({\mathrm{c-ind}}_{M^{\circ}}^M(\sigma_0)). \end{align*}$$

Here, $\sigma _0$ is any irreducible component of the (semisimple) restriction $\sigma |_{M^{\circ }}$ . We now obtain a functor from the category $\text {Rep}_{(M,\sigma )}(G)$ to the category of right ${\mathrm {End}}_G(\Gamma _{(M,\sigma )})$ -modules given by

$$\begin{align*}\pi \mapsto {\mathrm{Hom}}(\Gamma_{(M,\sigma)},\pi). \end{align*}$$

The fact that $\Gamma _{(M,\sigma )}$ is a projective generator implies that this is an equivalence of categories. This is [Reference Bernstein4, Lemma 22]; a detailed proof of this fact is also given in [Reference Roche, Cunningham and Nevins22, Theorem 1.5.3.1].

Given a Bernstein component attached to $\mathfrak {s} = (M,\sigma )$ , we use $\mathcal {H}_{\mathfrak {s}}$ to denote ${\mathrm {End}}_G(\Gamma _{\mathfrak {s}})$ and refer to it as the Hecke algebra attached to the component $\mathfrak {s}$ . Furthermore, for any $\pi \in \text {Rep}(G)$ , we let $\pi _{\mathfrak {s}}$ denote the corresponding $\mathcal {H}_{\mathfrak {s}}$ -module ${\mathrm {Hom}}(\Gamma _{\mathfrak {s}},\pi )$ .

Although we do not use it here, we point out that there is another highly useful approach to analyzing Bernstein components, based on the theory of types developed by Bushnell and Kutzko [Reference Bushnell and Kutzko8]. One can show that the Hecke algebra used by Bushnell and Kutzko is in fact isomorphic to the algebra $\mathcal {H}_{\mathfrak {s}}$ introduced above; we prove this fact in Appendix A. Therefore—for the purposes of this paper—the two approaches are equivalent.

2.5 Cuspidal representations

Here, we briefly recall some facts and introduce notation related to cuspidal representations of classical groups.

Let $\rho $ and $\tau $ be irreducible unitarizable cuspidal representations of $\mathrm {GL}_k(F)$ and $G_{n_0}$ , respectively. We consider the representation $\nu ^{\alpha }\rho \rtimes \tau $ , where $\alpha \in \mathbb {R}$ . Here, and throughout the paper, we use $\nu $ to denote the unramified character $\lvert \det \rvert $ of the general linear group. If $\rho $ is not self-dual, the above representation never reduces. If $\rho $ is self-dual, then there exists a unique $\alpha \geq 0$ such that $\nu ^{\alpha }\rho \rtimes \tau $ is reducible; we denote it by $\alpha _{\rho }$ .

The number $\alpha _{\rho }$ has a natural description in terms of Langlands parameters. Let $\phi $ be the L-parameter of $\tau $ . Then $\phi $ decomposes into a direct sum of irreducible representations of $W_F \times \text {SL}_2(\mathbb {C})$ . We view $\rho $ as a representation of $W_F$ ; we say that it is of the same type as $\phi $ if the corresponding $W_F$ -representation factors through a group of the same type (orthogonal/symplectic) as $\phi $ . Letting $S_a$ denote the (unique) irreducible algebraic a-dimensional representation of $\text {SL}_2(\mathbb {C})$ , we now set

$$\begin{align*}a_{\rho} = \max\{a: \rho \otimes S_a \text{ appears in }\phi\}. \end{align*}$$

If the above set is empty, we let

$$\begin{align*}a_{\rho} = \begin{cases} -1, \quad &\text{if } \rho \text{ is of the same type as }\phi,\\ 0, \quad &\text{otherwise}. \end{cases} \end{align*}$$

With this description of $a_{\rho }$ , we have $\alpha _{\rho } = \frac {a_{\rho }+1}{2}$ .

2.6 The structure of the Hecke algebra

We retain the notation $\rho , \tau $ , and $G_n$ from the previous subsection, and consider the cuspidal component $\mathfrak {s}$ attached to the representation

$$\begin{align*}\underbrace{\rho \otimes \dotsb \otimes \rho}_{n \text{ times}}\ \otimes\ \tau \end{align*}$$

of the Levi subgroup $M = \mathrm {GL}_k(F) \times \dotsb \times \mathrm {GL}_k(F) \times G_{n_0}$ in $G_{N}$ , where $N = nk+n_0$ . In the rest of the paper, we restrict our attention to cuspidal components of the above form. This does not present a significant loss of generality, since the Hecke algebra of a general cuspidal component is the product of algebras corresponding to the components described above. To simplify notation, we set $\mathcal {H} = \mathcal {H}_{\mathfrak {s}}$ .

The structure of the Hecke algebra $\mathcal {H}$ has been completely described by Heiermann [Reference Heiermann16, Reference Heiermann17]. In his work, Heiermann shows that $\mathcal {H}$ is a Hecke algebra with parameters (the type of the algebra and the parameters depending on the specifics of the given case). When the component in question is of the form described above, we have three distinct cases, which we now summarize. For basic definitions and results on Hecke algebras with parameters, we refer to the work of Lusztig [Reference Lusztig19].

In what follows, we let t denote the order of the (finite) group $\{\chi \in X^{\text {nr}}(\mathrm {GL}_k(F)): \rho \otimes \chi \cong \rho \}$ . In all three cases, the commutative algebra $\mathcal {A}$ (see Section 2.3) is a subalgebra of $\mathcal {H}$ . In the present setting, the rank of the free module $M^{\sigma }/M^{\circ }$ is equal to n. We can thus identify $\mathcal {A} \cong \mathbb {C}[M^{\sigma }/M^{\circ }]$ with the algebra of Laurent polynomials $\mathbb {C}[X_1^{\pm },\dotsc ,X_n^{\pm }]$ . We fix this isomorphism explicitly: For $i = 1, \dotsc , n$ , let $h_i$ be the element of M which is equal to $\text {diag}(\varpi ,1,\dotsc ,1)$ on the ith $\mathrm {GL}$ factor, and equal to the identity elsewhere. Then $X_i = b_{h_i}^t$ . The three cases are:

1. (i) No representation of the form $\rho \otimes \chi $ with $\chi \in X^{\text {nr}}(\mathrm{GL}_{k}(F))$ is self-dual.

   In this case, the algebra $\mathcal {H}$ is described by an affine Coxeter diagram of type $\tilde {A}_{n-1}$ with equal parameters t. In other words, it is isomorphic to the algebra $\mathcal {H}_n$ described in [Reference Chan and Savin11]: There are elements $T_1, \dotsc , T_{n-1}$ which satisfy the quadratic relation

   $$\begin{align*}(T_i + 1)(T_i - q^t) = 0, \quad i = 1,\dotsc, n-1 \end{align*}$$
   and commutation relations
   $$\begin{align*}T_i f - f^{s_i}T_i = (q^t-1) \frac{f - f^{s_i}}{1 - X_{i+1}/X_i}, \quad i = 1,\dotsc, n-1, \end{align*}$$
   where $f^{s_i}$ is obtained from $f \in \mathcal {A}$ by swapping $X_i$ and $X_{i+1}$ .

In the two remaining cases, there is an unramified character $\chi $ such that $\rho \otimes \chi $ is self-dual. Without loss of generality, we may assume that $\rho $ is self-dual. Then, up to isomorphism, there is a unique representation of the form $\rho \otimes \chi \ncong \rho $ which is also self-dual; we denote it by $\rho ^-$ . We set $\alpha = \alpha _{\rho }$ and $\beta = \alpha _{\rho ^-}$ (see Section 2.5 for notation). Since the situation is symmetric, we may (and will) assume that $\alpha \geq \beta $ . The description of $\mathcal {H}$ now involves two additional operators $T_0$ and $T_n$ (see Remark 2.1). We have the following two cases:

1. (ii) $\alpha = \beta = 0$ .

   In this case, $\mathcal {H}$ is described by an affine Coxeter diagram of type $\tilde {C}_n$ :

   
   The nodes correspond to operators $T_0, \dotsc , T_n$ which satisfy the quadratic relations
   $$\begin{align*}T_0^2 = 1, \quad T_n^2 = 1, \quad (T_i+1)(T_i-q^t) = 0\quad \text{for }i = 1, \dotsc, n-1, \end{align*}$$
   and the braid relations as prescribed by the diagram. The commutation relations for $T_i, i=1,\dotsc , n-1$ , are the same as in Case (i), whereas $T_n$ satisfies
   $$\begin{align*}fT_n - T_nf^{\vee} = 0 \end{align*}$$
   with $f^{\vee }(X_1,\dotsc ,X_{n-1},X_n) = f(X_1,\dotsc ,X_{n-1},1/X_n)$ .

2. (iii) $\alpha> 0$ .

   In this case, $\mathcal {H}$ is described by an affine Coxeter diagram of type $\tilde {C}_n$ :

   
   Here, $s = t(\alpha -\beta )$ and $r = t(\alpha + \beta )$ . Again, the nodes correspond to operators $T_0, \dotsc , T_n$ which satisfy quadratic relations analogous to those in Case (ii), along with the braid relations. The commutation relations for $T_i, i=1,\dotsc , n-1$ , are the same as in Case (i), whereas $T_n$ satisfies
   $$\begin{align*}fT_n - T_nf^{\vee} = \left((q^r-1) + \frac{1}{X_n}(\sqrt{q}^{r+s}-\sqrt{q}^{r-s})\right)\frac{f - f^{\vee}}{1-1/X_n^2}. \end{align*}$$

Cases (i)–(iii) correspond to Cases (I)–(III) listed in [Reference Heiermann16, Section 3.1]. The above results are collected in Section 3.4 of [Reference Heiermann16]. We take a moment to explain the situation in the even orthogonal case. Papers [Reference Heiermann16, Reference Heiermann17] do not treat the full orthogonal group; rather, they contain results about the special orthogonal group $\mathrm {SO}(2n)$ . In the special orthogonal case, there is a nontrivial R-group (see [Reference Goldberg14]) which complicates the structure of the Hecke algebra; this was ultimately worked out by Heiermann in [Reference Heiermann18]. Because of this, we choose to work with $\mathrm {O}(2n)$ instead. This is indeed justified: Annex A of [Reference Heiermann18] shows that the results of [Reference Heiermann16, Reference Heiermann17] generalize to the full orthogonal case.

A detailed construction of the operators $T_i$ (starting from standard intertwining operators) is the subject matter of [Reference Heiermann17]; we do not need the details here, except in a special case discussed in the final part of Section 3.2. To facilitate the comparison of the above summary to the works of Heiermann [Reference Heiermann16–Reference Heiermann18], we point out the ways in which our summary deviates from them.

2.7 Generic representations

We recall only the most basic facts here; a general reference is, e.g., [Reference Shahidi24].

Assume that G is split, and let U is be a maximal unipotent subgroup of G. Fix a nondegenerate character $\psi $ of U. Recall that a character of U is said to be nondegenerate if it is nontrivial on every root subgroup corresponding to a simple root. We say that a representation $(\pi ,V)$ of G is $\psi $ -generic if there exists a so-called Whittaker functional—that is, a linear functional $L: V\to \mathbb {C}$ such that

$$\begin{align*}L(\pi(u)v) = \psi(u)L(v), \quad \forall u\in U, v\in V. \end{align*}$$

The key fact we use throughout is that the space of Whittaker functionals is at most one-dimensional. However, this fact does not hold for the disconnected $\mathrm {O}(2n, F)$ , and we need to adjust the definition of Whittaker character as follows. In this case, the Levi factor of the normalizer of U in $\mathrm {O}(2n, F)$ is $\mathrm {GL}_1(F) \times \cdots \times \mathrm {GL}_1(F) \times \mathrm {O}(2, F)$ and there exists $\alpha \in \mathrm {O}(2,F) \setminus \mathrm {SO}(2,F)$ normalizing $\psi $ . Observe that the order of $\alpha $ is 2. We extend $\psi $ to a character $\tilde \psi $ of $\tilde U= U\rtimes \langle \alpha \rangle $ by $\tilde \psi (\alpha )=1$ . With this extension, the space of Whittaker functionals for any irreducible representation of $\mathrm {O}(2n, F)$ is at most one-dimensional.

Now, let $P=MN$ be a parabolic subgroup of G. If $\sigma $ is an irreducible generic representation of M, then one can construct a Whittaker functional on $i_P^G\sigma $ (see [Reference Shahidi24, Proposition 3.1] and equation (3.11)); in other words, the induced representation is $\psi $ -generic as well. We use this fact later, in Section 3.2.

3.1 $\mathcal {H}$ -module structures on $\mathcal {A}$

In order to treat the case of general Bernstein components—and not just those described in Section 2.6—we work in a slightly more general setting in this section. We thus investigate the possible $\mathcal {H}$ -module structures on $\mathcal {A}$ (where $\mathcal {H}$ is generated by $T_0,\dotsc , T_n$ over $\mathcal {A}$ ), but we assume that $\mathcal {A} = \mathcal {A}'[X_1^{\pm },\dotsc , X_n^{\pm }]$ , where $\mathcal {A}'$ is an integral domain containing $\mathbb {C}$ as a subring. For Bernstein components described in Section 2.6, we have $\mathcal {A}' = \mathbb {C}$ ; in general, $\mathcal {A}'$ itself is a (Laurent) polynomial ring over $\mathbb {C}$ .

First, assume that we are in Case (i) (see Section 2.6). Then the situation is precisely the one treated in [Reference Chan and Savin11], and the possible $\mathcal {H}$ -module structures on $\mathcal {A}$ are determined in Section 2.2 there. We have the following.

Here, $\mathcal {H}_{S_n}$ denotes the finite-dimensional algebra generated by $T_1,\dotsc ,T_{n-1}$ ; we have $\mathcal {H} = \mathcal {A} \otimes _{\mathbb {C}} \mathcal {H}_{S_n}$ . Furthermore, $\mathcal {H}_{S_n}$ has precisely two one-dimensional representations:

$$ \begin{align*} \epsilon_{-1}: T_i \mapsto -1 \quad &\text{ for } i = 1, \dotsc, n-1; \quad \text{and}\\ \epsilon_{q^t}: T_i \mapsto q^t \quad &\text{ for } i = 1, \dotsc, n-1. \end{align*} $$

We now treat Cases (ii) and (iii) simultaneously. Recall that, in these cases, the algebra $\mathcal {H}$ is described by an affine Coxeter diagram of type $\tilde {C}_n$ . We let $\mathcal {H}_0$ and $\mathcal {H}_n$ denote the algebras obtained by removing the vertices which correspond to $T_0$ and $T_n$ , respectively. In other words, $\mathcal {H}_0$ is generated by $T_1,\dotsc ,T_n$ as an $\mathcal {A}$ -algebra, whereas $\mathcal {H}_n$ is generated by $T_0,\dotsc ,T_{n-1}$ . Note that we have $\mathcal {H} = \mathcal {A} \otimes _{\mathbb {C}}\mathcal {H}_n = \mathcal {A} \otimes _{\mathbb {C}}\mathcal {H}_0$ . We now prove the following result.

Proposition 3.4 (Cases (ii) and (iii))

Let $\Pi $ be an $\mathcal {H}$ -module which is isomorphic to $\mathcal {A}$ as an $\mathcal {A}$ -module. Then

$$\begin{align*}\Pi \cong \mathcal{H} \otimes_{\mathcal{H}_{0}} \epsilon_0 \quad \text{or} \quad \Pi \cong \mathcal{H} \otimes_{\mathcal{H}_{n}} \epsilon_n. \end{align*}$$

Here, $\epsilon _0$ (resp. $\epsilon _n$ ) is a one-dimensional representation of $\mathcal {H}_{0}$ (resp. $\mathcal {H}_{n}$ ).

Proof We first restrict our attention to the subalgebra generated by $T_1,\dotsc , T_{n-1}$ , which is contained in both $\mathcal {H}_0$ and $\mathcal {H}_n$ . This is precisely the algebra $\mathcal {H}_{S_n}$ discussed in [Reference Chan and Savin11]. The possible $\mathcal {H}_{S_n}$ -structures on $\mathcal {A}$ are determined in Section 2.2 there. To summarize the relevant results, there exists an invertible element $g_0\in \mathcal {A}$ on which the operators $T_1,\dots ,T_n$ act by the same scalar, either $q^t$ or $-1$ .

We now determine how $T_0$ and $T_n$ act on $g_0$ . Since $g_0$ is invertible, we have $T_ng_0 = fg_0$ for some $f\in \mathcal {A}$ . Recall that $T_n$ satisfies the quadratic relation

$$\begin{align*}T_n^2 = (q^r-1)T_n + q^r \end{align*}$$

as well as the commutation relation

$$\begin{align*}T_nf- f^{\vee} T_n = \left((q^r-1) + \frac{1}{X_n}(\sqrt{q}^{r+s}-\sqrt{q}^{r-s})\right)\frac{f-f^{\vee}}{1-1/X_n^2}. \end{align*}$$

Here, and throughout the proof, we let $r=s=0$ if we are considering Case (ii). Recall that $f^{\vee }$ denotes the function $f^{\vee }(X_1,\dotsc ,X_n) = f(X_1,\dotsc ,X_{n-1},\frac {1}{X_n})$ . Using the above and comparing the two sides of $T_n^2g_0 = (q^r-1)T_ng_0 + q^rg_0$ , we get

$$\begin{align*}ff^{\vee} = (q^r-1)\frac{X_nf^{\vee}-\frac{1}{X_n}f}{X_n-\frac{1}{X_n}} - (\sqrt{q}^{r+s}-\sqrt{q}^{r-s})\frac{f-f^{\vee}}{X_n-\frac{1}{X_n}} + q^r. \end{align*}$$

To simplify notation, we now set $b = q^r-1$ and $c = (\sqrt {q}^{r+s}-\sqrt {q}^{r-s})$ . We also temporarily drop the index n, writing X instead of $X_n$ . Clearing out the denominators, we rearrange the above equation into

(*) $$ \begin{align} (X^2-1)ff^{\vee} = b(X^2f^{\vee} - f) - c(Xf- Xf^{\vee}) + q^r(X^2-1). \end{align} $$

Our first goal is to find the possible solutions $f\in \mathcal {A}$ of this equation.

Lemma 3.5 The above equation has the following solutions:

(i) $$ \begin{align} f &= b + cX^{-1} + bX^{-2} + \cdots + cX^{1-2d} +q^tX^{-2d}, && \quad \ \ \ \ \, d \in \mathbb{Z}_{>0} \end{align} $$

(ii) $$ \begin{align} f &= b + cX^{-1} + bX^{-2} + \cdots + cX^{1-2d} - X^{-2d}, \, && \qquad \ \ \ d \in \mathbb{Z}_{>0} \end{align} $$

(iii) $$ \begin{align} f &= b + cX^{-1} + bX^{-2} + \cdots + bX^{-2d} \pm \sqrt{q}^{r\pm s}X^{-2d-1}, && d \in \mathbb{Z}_{\geq 0} \end{align} $$

(iv) $$ \begin{align} f &= \mp\sqrt{q}^{r\pm s}X^{2d+1}-bX^{2d} -cX^{2d-1}-\dotsb-cX, && \qquad\ d \in \mathbb{Z}_{\geq 0} \end{align} $$

(v) $$ \begin{align} f &= -q^rX^{2d} - cX^{2d-1} - \cdots - bX^2-cX, \ \ && \qquad\qquad \ \ \ d \in \mathbb{Z}_{>0} \end{align} $$

(vi) $$ \begin{align} f &= X^{2d} - cX^{2d-1} - \dotsb - bX^2-cX, \ \, && \qquad\qquad\qquad \ \, d \in \mathbb{Z}_{>0} \end{align} $$

along with the constant solutions $f = q^t$ and $f=-1$ .

Proof Each $f\in \mathcal {A}$ can be written as

(†) $$ \begin{align} f = a_kX^k + a_{k-1}X^{k-1} + \dotsb + a_0 + a_{-1}X^{-1} + \dotsb + a_{-l}X^{-l} \end{align} $$

for some functions $a_{-l},\dotsc ,a_k \in \mathcal {A'}[X_1^{\pm }, \dotsc , X_{n-1}^{\pm }]$ , with $a_k, a_{-l}\neq 0$ . We write $\text {maxdeg}(f)$ for k and $\text {mindeg}(f)$ for $-l$ . Now, let f be a solution of (*). We begin our analysis of (*) by solving some special cases. We claim the following:

(3.1) $$ \begin{align} \quad \begin{matrix} \text{If} & f =a_0, &\text{then} & a_0 = q^r \text{ or } a_0 = -1.\\ \text{If} & f =a_1X, &\text{then} & a_1 = \mp\sqrt{q}^{r\pm s}.\\ \text{If} & f = a_0 + a_{-1}X^{-1} \text{ and } a_{-1}\neq 0, &\text{ then } & a_0 = b \text{ and } a_{-1} = \pm\sqrt{q}^{r\pm s}. \end{matrix} \end{align} $$

To verify this, we first look at solutions $f = a_0$ . In this case, the equation (*) reduces to $a_0^2 = ba_0 + q^r$ . This equation has two constant solutions, $a_0 = -1$ and $a_0 = q^r$ . These are also the only solutions, since $\mathcal {A}$ has no zero divisors. When $f(X) = a_1X$ , the equation becomes $a_1^2 + a_1c - q^r = 0$ . Again, the only two solutions of this equation are the constant ones: $a_1 = \mp \sqrt {q}^{r\pm s}$ . Finally, when $f = a_0 +a_{-1}X^{-1}$ , the equation reduces to the following system:

$$\begin{align*}a_1b = a_1a_0\quad \text{and}\quad a_0^2 + a_{-1}^2 = a_0b + a_{-1}c + q^r. \end{align*}$$

Since we are assuming that $a_1\neq 0$ , the first equation gives us $a_0 = b$ , and then the second becomes $a_{-1}^2 - ca_{-1} - q^r = 0$ . Again, we have two solutions: $a_{-1} = \pm \sqrt {q}^{r\pm s}$ .

Next, when f is a solution of (*) given by (†), we observe

(3.2) $$ \begin{align} k \text{ and } l \text{ cannot both be positive.} \end{align} $$

Indeed, let LHS and RHS denote the left-hand side and the right-hand side of (*), respectively. We then have $\mathrm{maxdeg(LHS)} = k+l+2$ , whereas $\mathrm{maxdeg(RHS)} \leq \max \{l+2,k+1,2\}$ . Therefore, equality of degrees cannot be achieved unless $k \leq 0$ or $l\leq 0$ . In fact, the same argument gives us a slightly stronger statement in one case:

(3.3) $$ \begin{align} \text{If } k> 0, \text{ then } a_0 = 0. \end{align} $$

Finally, we make use of the following fact, which is readily verified by direct computation:

(3.4) $$ \begin{align} \begin{aligned} &\text{For any positive integer } d, f \text{ is a solution of (*) if and only if }\\ &\qquad\qquad\qquad\qquad X^{2d}f-R_d \text{ is also a solution.} \end{aligned} \end{align} $$

Here, $R_d = \dfrac {bX^2+cX}{X^2-1}(X^{2d}-1) = bX^{2d} + cX^{2d-1}+\cdots +bX^2+cX$ .

We are now ready to find all the solutions. By (3.2), any solution of f contains either only positive powers of X, or only nonpositive. We therefore consider two separate cases.

Case A: f has only nonpositive powers, i.e., $f = a_0 + a_{-1}X^{-1} + \dotsb + a_{-l}X^{-l}$ .

Let $d = \lfloor l/2 \rfloor $ . We use (3.4) and look at another solution, $g = X^{2d}f - R_d$ .

We first assume that $l=2d$ is even. In this case, g only has nonnegative powers of X, but it has a nonzero constant term, $a_{-l}$ . Therefore, (3.3) shows that the coefficients next to the positive powers must be zero: $a_0-b = a_{-1}-c= \dotsb = a_{-l+1}-c = 0$ . Now, (3.1) shows that there are only two possibilities for the constant term: $a_{-l} = q^t$ or $a_{-l} = -1$ . We thus get two solutions:

$$\begin{align*}&f = b + cX^{-1} + bX^{-2} + \cdots + cX^{1-2d} +q^tX^{-2d} \text{ } \text{ and } \\ & f = b + cX^{-1} + bX^{-2} + \cdots + cX^{1-2d} - X^{-2d}. \end{align*}$$

Next, assume that $l=2d+1$ is odd. Now, g has a nonzero coefficient (i.e., $a_{-l}$ ) next to $X^{-1}$ , so by (3.2) the coefficients next to positive powers must be equal to $0$ . This gives us $a_0 = b, a_{-1} = c, \dotsc , a_{2-l} = c$ . Furthermore, g is thus of the form $a_{1-l} + a_{-l}X^{-1}$ , so we can read off the coefficients $a_{1-l}$ and $a_{-l}$ from (3.1). We thus arrive at two more solutions:

$$\begin{align*}f = b + cX^{-1} + bX^{-2} + \cdots + bX^{-2d} \pm \sqrt{q}^{r\pm s}X^{-2d-1}. \end{align*}$$

Case B: f only has positive powers, i.e., $f = a_kX^k +\cdots + a_1X$ .

This time, we set $d = \lfloor k/2 \rfloor $ and use (3.4) to obtain the solution $g = \dfrac {1}{X^{2d}}(f+R_d)$ .

First, assume that $k = 2d+1$ is odd. Then g has a nonzero coefficient (i.e., $a_k$ ) next to X, so (3.2) and (3.3) imply that all the lower coefficients are zero. This immediately gives us $a_1 = -c, a_2 = -b, \dotsc , a_{2d} = -b$ . Furthermore, we have $g = a_kX$ , so (3.1) shows that we have two possibilities for $a_k$ . We therefore get two solutions:

$$\begin{align*}f = \mp\sqrt{q}^{r\pm s}X^{2d+1}-bX^{2d} -cX^{2d-1}-\dotsb-cX. \end{align*}$$

Finally, assume that $k=2d$ is even. First, if $k> 2$ , consider another solution $g' = X^{2-2d}(f+R_{2d-2})$ . Now, $g'$ has a nonzero coefficient (i.e., $a_k$ ) next to $X^2$ , so the coefficient next to nonpositive powers of X have to be $0$ by (3.2) and (3.3). This gives us $a_1 = -c, a_2 = -b, \dotsc , a_{2d-2}= -b$ . In particular, this shows that $g = (a_k+b) + (a_{k-1}+c)X^{-1}$ . Since $a_k + b \neq b$ (i.e., $a_k\neq 0$ ), (3.1) shows that we have only two possibilities:

$$\begin{align*}a_{k-1} +c = 0 \quad \text{and} \quad a_k+b \in \{q^r, -1\}. \end{align*}$$

In other words, $a_{k-1} = -c$ and $a_k \in \{-q^r, 1\}$ . We thus get the remaining solutions:

$$\begin{align*}f = -q^rX^{2d} - cX^{2d-1} - \cdots - bX^2-cX \quad \text{and}\quad f = X^{2d} - cX^{2d-1} - \cdots - bX^2-cX. \end{align*}$$

We continue the proof of Proposition 3.4. We have just proved that $T_ng_0 = fg_0$ where $f\in \mathcal {A}$ is one of the elements listed in Lemma 3.5. First, assume that f is one of the constant solutions, i.e., $f=-1$ or $f=q^r$ . Then $g_0$ is an invertible element of $\mathcal {A}$ on which $T_1, \dotsc , T_{n-1}, T_n$ all act as scalars. In other words, we have a one-dimensional representation $\epsilon _0$ of the algebra $\mathcal {H}_0$ . Since $\mathcal {H} = \mathcal {A} \otimes _{\mathbb {C}} \mathcal {H}_0$ , it follows that the corresponding $\mathcal {H}$ -module structure on $\mathcal {A}$ is isomorphic to

$$\begin{align*}\mathcal{H}\otimes_{ \mathcal{H}_0} \epsilon_0. \end{align*}$$

Now, if f is of type (i) or (ii) listed in the statement of Lemma 3.5, set

$$\begin{align*}g_1 = (X_1X_2\cdot \dotsb \cdot X_n)^{-d}g_0. \end{align*}$$

Since $(X_1X_2\cdot \dotsb \cdot X_n)^{-d}$ commutes with $T_1,\dotsc ,T_{n-1}$ , $g_1$ is still an eigenvector for each of these operators. We claim that $g_1$ is an eigenvector for $T_n$ as well. Indeed, using the appropriate commutation relation and the fact that $T_n$ commutes with $X_1, \dotsc , X_{n-1}$ , we get

$$ \begin{align*} T_ng_1 &= (X_1X_2\cdot \dotsb \cdot X_{n-1})^{-d}\cdot T_nX_n^{-d}g_0\\ &= (X_1X_2\cdot \dotsb \cdot X_{n-1})^{-d}\left(X_n^dT_n + \frac{bX_n+c}{X_n^2-1}(X_n^{-d}-X_n^d)\right)g_0\\ &= (X_1X_2\cdot \dotsb \cdot X_{n-1})^{-d}\left(X_n^df + \frac{bX_n+c}{X_n^2-1}(X_n^{-d}-X_n^d)\right)g_0\\ &= (X_1X_2\cdot \dotsb \cdot X_{n-1})^{-d}\left(X_n^{2d}f - \frac{bX_n+c}{X_n^2-1}(X_n^{2d}-1)\right)g_0. \end{align*} $$

Simplifying the expression in the parentheses, we obtain $\lambda X_n^{-d}$ , so that $T_ng_1 = \lambda g_1$ , where $\lambda = q^t$ (resp. $-1$ ) when f is of type (i) (resp. (ii)). We have thus once more found a common eigenvector for $T_1,\dotsc ,T_{n-1},T_n$ . Again, we deduce that the corresponding $\mathcal {H}$ -module structure is isomorphic to $\mathcal {H}\otimes _{ \mathcal {H}_0} \epsilon _0$ , where $\epsilon _0$ is a one-dimensional representation of $\mathcal {H}_0$ .

When f is of type (v) or (vi), we use the same argument and arrive at the same conclusion. The only difference in this case is that we have to set $g_1 = (X_1X_2\cdot \dotsb \cdot X_n)^{d}g_0$ in order to obtain a common eigenvector for $T_1,\dotsc ,T_{n-1},T_n$ .

In the remaining cases—that is, when f is of type (iii) or (iv)—we cannot find such an eigenvector, but we claim that we can find an invertible $g_1 \in A$ which is a common eigenvector for $T_0, T_1, \dotsc , T_{n-1}$ . Just like in the previous cases, this will imply that the $\mathcal {H}$ -structure on $\mathcal {A}$ is isomorphic to $\mathcal {H}\otimes _{ \mathcal {H}_n} \epsilon _n$ for some one-dimensional representation $\epsilon _n$ of $\mathcal {H}_n$ .

If $T_ng_0 = fg_0$ with f of type (iii), we set $g_1 = (X_1X_2\cdot \dotsb \cdot X_n)^{-d}g_0$ . If f is of type (iv), let $g_1 = (X_1X_2\cdot \dotsb \cdot X_n)^{d+1}g_0$ . In both cases, $g_1$ is an eigenvector for $T_1,\dotsc , T_{n-1}$ and a computation analogous to the one we carried out in for Cases (i) and (ii) shows that we have

$$\begin{align*}T_ng_1 = (b\pm \sqrt{q}^{r\pm s}X_n^{-1})g_1. \end{align*}$$

The following lemma then shows that $g_1$ is also an eigenvector for $T_0$ and thus concludes the proof of Proposition 3.4.

Lemma 3.6 Let g be an invertible element of $\mathcal {A}$ , which is an eigenvector for $T_1, \dotsc , T_{n-1}$ , such that $T_ng = (b\pm \sqrt {q}^{r\pm s}X_n^{-1})g$ . Then g is also an eigenvector for $T_0$ .

Proof Recall that $T_0 = \sqrt {q}^{s+2(n-1)t+r}X_1T_w^{-1}$ , with $T_w = T_1\dotsm T_{n-1}T_nT_{n-1}\dotsm T_1$ . In both cases, all the operators $T_1,\dotsc ,T_{n-1}$ act by the same scalar $\lambda \in \{-1,q^t\}$ . We therefore have

$$\begin{align*}T_0g = \sqrt{q}^{s+2(n-1)t+r} \lambda^{-(n-1)} X_1 T_1^{-1}\cdot\dotsb\cdot T_{n-1}^{-1}T_n^{-1}g. \end{align*}$$

We now recall that $T_n^{-1} = \frac {1}{q^r}(T_n-b)$ ; this follows from the quadratic relation for $T_n$ . Therefore, by the assumption in the statement of the lemma, $T_n^{-1}g = \pm \sqrt {q}^{\pm s -r}X_n^{-1}$ . Thus,

(3.5) $$ \begin{align} T_0g = \mu\cdot \lambda^{-(n-1)} \cdot \sqrt{q}^{2(n-1)t}X_1 T_1^{-1}\cdot\dotsb\cdot T_{n-1}^{-1}X_n^{-1}g, \end{align} $$

with $\mu \in \{-1, q^s\}$ . Finally, it remains to notice that, for every $i = 1, \dotsc , n-1$ , we have

(3.6) $$ \begin{align} T_i^{-1}X_{i+1}^{-1} = \frac{1}{q^t}X_{i}^{-1}T_{i}. \end{align} $$

Indeed, from the quadratic relation, we have $T_i^{-1}= \frac {1}{q^t}(T_i - (q^t-1))$ . Combining this with the commutation relation for $T_i$ , we get (3.6). Successively applying (3.6) to (3.5) (and taking into account that each $T_i$ acts on g by $\lambda $ ), we get

$$\begin{align*}T_0g = \mu g, \end{align*}$$

which we needed to prove. Notice that the possible eigenvalues are precisely the zeros of $(x-q^s)(x+1) = 0$ , the quadratic equation satisfied by $T_0$ .

The above lemma shows that, in Cases (iii) and (iv), we have an invertible element $g_1 \in \mathcal {A}$ which is a common eigenvector for $T_0,T_1,\dotsc , T_n$ . Consequently, the $\mathcal {H}$ -module structure on $\mathcal {A}$ is given by $\mathcal {H}\otimes _{ \mathcal {H}_n} \epsilon _n$ for some one-dimensional representation $\epsilon _n$ of $\mathcal {H}_n$ . This concludes the proof of Proposition 3.4.

In view of Proposition 3.4, there are eight candidates for the $\mathcal {H}$ -structure (four, if $n=1$ ): First, we may take the tensor product over $\mathcal {H}_0$ or $\mathcal {H}_n$ ; after that, there are four one-dimensional representations of $\mathcal {H}_0$ (resp. $\mathcal {H}_n$ ) to choose from. To verify this, note that the braid relations imply that—in any one-dimensional representation—the operators $T_1, \dotsc , T_{n-1}$ act by the same scalar, which has to be a zero of the quadratic relation satisfied by $T_i$ : $ (x-q^t)(x+1) = 0$ . We therefore have two possibilities for the action of the operators $T_i$ , and two additional possibilities (again, the zeros of the quadratic relation) for $T_n$ (resp. $T_0$ ). For example, the one-dimensional representations of $\mathcal {H}_0$ are given by

$$ \begin{align*} \epsilon_{-1,-1} &: \{T_n \mapsto -1, T_i \mapsto -1\}, \quad &\epsilon_{q^r,-1} : \{T_n \mapsto q^r, T_i \mapsto -1\},\\ \epsilon_{-1,q^t} &: \{T_n \mapsto -1, T_i \mapsto q^t\}, \quad &\epsilon_{q^r,q^t} : \{T_n \mapsto q^r, T_i \mapsto q^t\}. \end{align*} $$

Corollary 3.7 General case. Let $\Pi $ be an $\mathcal {H}$ -module which is isomorphic to $\mathcal {A}$ as an $\mathcal {A}$ -module. Then there exists a finite subalgebra ${\mathcal H}_W \cong {\mathbb C}[W]$ , where W is a finite group, such that ${\mathcal H} \cong \mathcal A \otimes {\mathcal H}_W$ , and

$$\begin{align*}\Pi \cong \mathcal{H} \otimes_{\mathcal{H}_{W}} \epsilon, \end{align*}$$

where $\epsilon $ is a one-dimensional representation of $\mathcal {H}_{W}$ .

3.2 The Gelfand–Graev module

To complete the analysis of the Gelfand–Graev representation, we need to determine which of the $\mathcal {H}$ -module structures from the previous section is isomorphic to $\Pi = ({\mathrm {c-ind}}_U^G(\psi ))_{\mathfrak {s}}$ . We consider Cases (i)–(iii) separately.

Case (i). Let $\delta $ be the unique irreducible subrepresentation of $\rho \nu ^{\frac {n-1}{2}} \times \rho \nu ^{\frac {n-3}{2}} \times \dotsb \times \rho \nu ^{\frac {1-n}{2}}$ . Then $\pi = \delta \rtimes \tau $ is an irreducible generic representation. The corresponding $\mathcal {H}$ -module is one-dimensional: By the Bernstein version of Frobenius reciprocity, we have

(3.7) $$ \begin{align} \begin{aligned} {\mathrm{Hom}}_G(\Gamma_{\mathfrak{s}}, \pi) = {\mathrm{Hom}}_M({\mathrm{c-ind}}_{M^{\circ}}^M(\rho \otimes \dotsb \otimes \rho \otimes \tau), &\ \nu^{\frac{1-n}{2}}\rho \otimes \dotsb \otimes \nu^{\frac{n-1}{2}}\rho\otimes \tau\\ \oplus &\ \nu^{\frac{1-n}{2}}\rho^{\vee} \otimes \dotsb \otimes \nu^{\frac{n-1}{2}}\rho^{\vee} \otimes \tau). \end{aligned} \end{align} $$

Since $\rho ^{\vee }$ is not an unramified twist of $\rho $ in this case, the above ${\mathrm {Hom}}$ -space is only one-dimensional. By Proposition 3.3, ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \Pi )$ is isomorphic to either $\Pi \cong \mathcal {H} \otimes _{\mathcal {H}_{S_n}} \epsilon _{-1}$ or $\Pi \cong \mathcal {H} \otimes _{\mathcal {H}_{S_n}} \epsilon _{q^t}$ . To determine which, we need only look at the action of $\mathcal {H}$ on the one-dimensional module $\pi $ . We now need to examine the definition of the operators $T_i, i= 1,\dotsc , {n-1}$ . In [Reference Heiermann17], $T_i$ is defined in Section 5.2 by the formula

(3.8) $$ \begin{align} T_i = R_i + (q^t-1)\dfrac{X_i/X_{i+1}}{X_{i}/X_{i+1}-1}. \end{align} $$

The intertwining operator $R_i$ has a pole at $0$ , and a zero at the point of reducibility (see [Reference Heiermann17, Section 1.8]). Since $\nu ^{\frac {3-n}{2}-i}\rho \times \nu ^{\frac {3-n}{2}-i+1}\rho $ reduces, the operator $R_i$ acts by $0$ in this case. It therefore remains to determine the action of $X_i/X_{i+1}$ . Equation (3.7) shows that it suffices to determine the action of $X_i/X_{i+1}$ on

$$\begin{align*}{\mathrm{Hom}}_M({\mathrm{c-ind}}_{M^{\circ}}^M(\rho \otimes \dotsb \otimes \rho \otimes \tau), \nu^{\frac{1-n}{2}}\rho \otimes \dotsb \otimes \nu^{\frac{n-1}{2}}\rho\otimes \tau). \end{align*}$$

Recalling the definition of $X_i$ (Section 2.6), we immediately see that $X_i/X_{i+1}$ acts by

$$\begin{align*}\dfrac{(|\varpi|^{\frac{3-n}{2}-i})^t}{(|\varpi|^{\frac{3-n}{2}-i+1})^t} = \frac{q^{t(\frac{n-3}{2}+i)}}{q^{t(\frac{n-3}{2}+i-1)}} = q^t. \end{align*}$$

This implies that $T_i$ also acts by $(q^t-1)\dfrac {q^t}{q^t-1} = q^t$ . Since $\pi $ is a quotient of $\Pi $ , we conclude that we must have $\Pi \cong \mathcal {H} \otimes _{\mathcal {H}_{S_n}} \epsilon _{q^t}$ .

Case (iii). In this situation, the $\mathfrak {s}$ -component of the Gelfand–Graev representation has two irreducible generic representations whose $\mathcal {H}$ -module is one-dimensional. These are the two (generalized) Steinberg representations: $\pi $ and $\pi '$ , which are the unique irreducible subrepresentations of

$$\begin{align*}\nu^{\alpha+n-1}\rho \times \dotsb \times \nu^{\alpha}\rho \rtimes \tau\quad \text{and}\quad \nu^{\beta+n-1}\rho^- \times \dotsb \times \nu^{\beta}\rho^- \rtimes \tau, \end{align*}$$

respectively. Recall that $\alpha $ (resp. $\beta $ ) is the unique positive real number such that $\nu ^{\alpha }\rho \rtimes \tau $ (resp. $\nu ^{\beta }\rho ^{-} \rtimes \tau $ ) reduces (see Section 2.6). We now compare the action of the operators $T_0, \dotsc , T_n$ on these two representations—that is, on ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \pi )$ and ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \pi ^-)$ , where $\Gamma _{\mathfrak {s}}$ is the projective generator defined in Section 2.4.

We start by analyzing the action on $\pi $ . We first focus on $T_i, i = 1, \dotsc , n-1$ . Again, $T_i$ is defined by (3.8), and once more, the operator $R_i$ acts by $0$ . By the Bernstein version of Frobenius reciprocity, we have

$$\begin{align*}{\mathrm{Hom}}_G(\Gamma_{\mathfrak{s}}, \pi) = {\mathrm{Hom}}_M({\mathrm{c-ind}}_{M^{\circ}}^M(\rho \otimes \dotsb \otimes \rho \otimes \tau), \nu^{-\alpha-{n+1}}\rho \otimes \dotsb \otimes \nu^{-\alpha}\rho\otimes \tau). \end{align*}$$

We immediately see that $X_i/X_{i+1}$ acts by

$$\begin{align*}\dfrac{(|\varpi|^{-\alpha-n+i})^t}{(|\varpi|^{-\alpha-n+i+1})^t} = \frac{q^{t(\alpha+n-i)}}{q^{t(\alpha+n-i-1)}} = q^t. \end{align*}$$

Again, this shows that $T_i$ acts by $(q^t-1)\dfrac {q^t}{q^t-1} = q^t$ . For $T_n$ , we have a similar formula:

(3.9) $$ \begin{align} T_n = R_n + (q^r-1)\dfrac{X_n\left(X_n-\dfrac{q^{t\beta}-q^{t\alpha}}{q^r-1}\right)}{X_n^2-1}. \end{align} $$

Once more, $R_n$ acts by $0$ , and $X_n$ acts by $(|\varpi |^{-\alpha })^t = q^{t\alpha }$ . Recalling that $r=t(\alpha +\beta )$ , we see that $T_n$ acts by $q^r$ . Finally, since

$$\begin{align*}T_0 = \sqrt{q}^{r+2t(n-1)+s}X_1T_1^{-1}\cdot \dotsb \cdot T_{n-1}^{-1}T_n^{-1}T_{n-1}^{-1}\cdot \dotsb \cdot T_1^{-1}, \end{align*}$$

and since $X_1$ acts by $q^{(\alpha +n-1)t}$ , we see that $T_0$ acts by $\dfrac {\sqrt {q}^{r+2t(n-1)+s}}{q^{2t(n-1)}\cdot q^r}q^{(\alpha +n-1)t} =q^s$ .

We do the same with $\pi ^-$ . Again, $X_i/X_{i+1}$ acts by $q^t$ , which shows that $T_i$ acts by $q^t$ as well. This time $X_n$ acts by $-q^{t\beta }$ : Recall that $\rho ^{-} = \chi _0 \otimes \rho $ with $X_n(\chi _0) = -1$ , so $X_n(\chi _0\nu ^{-\beta })= -q^{t\beta }$ . Repeating the above calculations, we now see that $T_n$ acts by $q^r$ , whereas $T_0$ acts by $-1$ .

The above analysis allows us to single out the $\mathcal {H}$ -module structure on $\Pi $ . Since $T_0$ does not act by the same scalar on $\pi $ and $\pi ^-$ , we deduce that $\Pi = \mathcal {H} \otimes _{\mathcal {H}_0} \epsilon $ for some one-dimensional representation $\epsilon $ of $\mathcal {H}_0$ . Now, since every $T_i$ ( $i=1,\dotsc ,n-1$ ) acts by $q^t$ and $T_n$ acts by $q^r$ , we deduce that $\Pi = \mathcal {H} \otimes _{\mathcal {H}_0} \epsilon _{q^r,q^t}$ (see the end of Section 3.1 for notation).

Case (ii) The first part of our analysis remains the same as in Case (iii). The representation

$$\begin{align*}\nu^{n-1}\rho \times \nu^{n-2}\rho \times \dotsb \times \rho \rtimes \tau \end{align*}$$

has two irreducible subrepresentations (both of which are in discrete series when $n>1$ , and tempered when $n=1$ ), only one of which is generic. Denote the generic subrepresentation by $\pi $ . Let $\pi ^-$ denote the generic representation resulting from an analogous construction, when $\rho $ is replaced by $\rho ^-$ . Again, the $\mathcal {H}$ -modules corresponding to $\pi $ and $\pi ^-$ are one-dimensional, and the same calculations we used in Case (iii) show that the operators $T_i$ , $i = 1, \dotsc , n-1$ , act by $q^t$ . This leaves us four possible $\mathcal {H}$ structures to consider

(3.10) $$ \begin{align} \qquad \begin{aligned} \mathcal{H} \otimes_{\mathcal{H}_0} \epsilon_0, \quad \text{with} \quad &\epsilon_0(T_n) = \pm 1 \quad (\text{and }\epsilon_0(T_i) = q^t,i = 1, \dotsc, n-1); \quad \text{and}\\ \mathcal{H} \otimes_{\mathcal{H}_n} \epsilon_n, \quad \text{with} \quad &\epsilon_n(T_0) = \pm 1 \quad (\text{and }\epsilon_n(T_i) = q^t, i = 1, \dotsc, n-1). \end{aligned} \end{align} $$

So far, we have been able to view Case (ii) as a special instance of Case (iii) which occurs when $r=s=0$ . However, to obtain an explicit description of the Gelfand–Graev module, we need more information than we used above in Case (iii). The reason is that the standard intertwining operator $\chi \rho \rtimes \tau \to \chi ^{-1}\rho ^{\vee } \rtimes \tau $ no longer has a pole when $X_n(\chi ) =\pm 1$ . In Case (iii), the operator $R_n$ (see formula (3.9))—which is constructed from the standard intertwining operator—vanishes at the point of reducibility, and the action of $T_n$ is determined by the action of the function

$$\begin{align*}(q^r-1)\dfrac{X_n\left(X_n-\dfrac{q^{t\beta}-q^{t\alpha}}{q^r-1}\right)}{X_n^2-1} \end{align*}$$

used to remove the poles of $R_n$ . In this case, however, $R_n$ no longer vanishes and is regular at the point of reducibility; consequently, the above function does not appear in the construction and we have $T_n = R_n$ . We know that this operator acts by $1$ or $-1$ on the $\mathcal {H}$ -modules $\pi $ and $\pi ^-$ , but we still have a certain amount of freedom in our choices. Indeed, as one verifies easily, the operator $T_n' = (-1)^eX_n^fT_n$ (where $e \in \{0,1\}$ and $f \in \mathbb {Z}$ ) satisfies the same relations as $T_n$ . Therefore, we obtain the same Hecke algebra if we replace $T_n$ by $T_n'$ , but the action of $T_n'$ on $\pi $ obviously differs from the action of $T_n$ .

In fact, we know that $X_n$ acts on $\pi $ by $1$ , and on $\pi ^-$ by $-1$ . Therefore, $X_n^2$ acts by $1$ on both, so replacing $T_n$ by $X_n^2T_n$ does not affect our description of the Gelfand–Graev module. We thus have four choices that affect the description ( $e = 0$ or $1$ ; f even or odd), and as we vary the four choices, the description of the Gelfand–Graev module varies through all the four possibilities described in (3.10).

This discussion shows that—to determine the action explicitly—we need to specify the choices appearing in the construction of the operator $R_n$ . We now explain one possible normalization using Whittaker models. To be concrete, we now focus on $G=\mathrm {SO}(2N+1)$ ; the same approach is possible when G is symplectic or even orthogonal. We also specialize our discussion to the case $n=1$ to simplify notation (thus, the cuspidal representation which defines the component is $\rho \otimes \tau $ ); the general case is analogous and follows from this one. We thus drop the subscripts and write $T,X$ instead of $T_n, X_n$ .

We fix a nondegenerate character $\psi $ of the unipotent radical U of $G = \mathrm {SO}(2N+1)$ . Let $V_{\rho }$ denote the space of the representation $\rho $ , and let $\lambda $ be a $\psi $ -Whittaker functional on $V_{\rho }$ : $\lambda (\rho (u)v) = \psi (u)\lambda (v)$ , for $v \in V_{\rho }$ . Notice that $\lambda $ is then also a $\psi $ -Whittaker functional for $\rho \otimes \chi $ for any unramified character $\chi \in \mathrm {GL}_k(F)$ : We have

$$\begin{align*}\lambda((\chi\otimes \rho)(u)v) = \chi(u)\psi(u)\lambda(v) = \psi(u)\lambda(v), \end{align*}$$

since $\det u = 1$ and thus $u \in \ker \chi $ . Abusing notation, we also let $\lambda $ denote the $\psi $ -Whittaker functional of $\rho \otimes \tau $ (or $\chi \rho \otimes \tau $ for any unramified $\chi $ , as we have just shown). Following Proposition 3.1 of [Reference Shahidi24], we now form a $\psi $ -Whittaker functional $\Lambda _{\chi }$ on the space of $i_P^G(\chi \rho \otimes \tau )$ by setting

(3.11) $$ \begin{align} \Lambda_{\chi}(f) = \int_N \lambda\left(f(wn)\right)\psi(n)^{-1} dn, \end{align} $$

where w is a representative of the nontrivial element of the Weyl group; in our case, we take w to be the block antidiagonal matrix

$$\begin{align*}\begin{pmatrix} \mbox{ } & \mbox{ } & I_k\\ \mbox{ } & I_{2(N-k)+1} & \mbox{}\\ I_k & \mbox{ } & \mbox{} \end{pmatrix}. \end{align*}$$

Since $\pi $ and $\pi ^-$ are generic, it suffices to determine the action of T on their respective Whittaker functionals if we want to determine how T acts on the $\mathcal {H}$ -modules ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \pi )$ and ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \pi ^-)$ .

For any unramified character $\chi $ , we have the specialization map ${\text {sp}_{\chi } : \Gamma _{\mathfrak {s}} \mapsto i_P^G(\chi \rho \otimes \tau )}$ (cf. [Reference Heiermann17, Section 3.1]). The unique (up to scalar multiple) element of ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \pi )$ factors through $\text {sp}_1 : \Gamma _{\mathfrak {s}} \to i_P^G(\rho \otimes \tau )$ ; similarly, any element of ${\mathrm {Hom}}_G(\Gamma _{\mathfrak {s}}, \pi ^-)$ factors through $\text {sp}_{\chi _0}$ (recall that $\rho ^- = \chi _0\otimes \rho $ ). Notice that $\Lambda _1$ and $\Lambda _{\chi _0}$ are the Whittaker models of $\pi $ and $\pi ^-$ , respectively.

To determine the action of T on $\Lambda _{\chi }$ (for any $\chi $ ), we must compare $\Lambda _{\chi }\text {sp}_{\chi }$ and $\Lambda _{\chi } \circ \text {sp}_{\chi } \circ T$ . The operator T is defined by the following property:

$$\begin{align*}\text{sp}_{\chi} T = \varphi \circ J(\chi^{-1}) \circ \text{sp}_{\chi^{-1}} \end{align*}$$

(cf. [Reference Heiermann17, Sections 3.1 and 3.2]). Here, $J(\chi ^{-1})$ denotes the standard intertwining operator $i_P^G(\chi ^{-1}\rho \otimes \tau ) \to i_P^G(\chi \rho ^{\vee } \otimes \tau )$ . To explain $\varphi $ , recall that $\rho $ is assumed to be self-dual. Therefore, we can fix an isomorphism $\varphi : \rho ^{\vee } \mapsto \rho $ and induce to an isomorphism $i_P^G(\chi \rho ^{\vee } \otimes \tau ) \to i_P^G(\chi \rho \otimes \tau )$ for any unramified $\chi $ , which we again denote by $\varphi $ by abuse of notation.

Let $\Lambda _{\chi }^{\vee }$ denote the Whittaker functional on $i_P^G(\chi \rho ^{\vee } \otimes \tau )$ obtained using (3.11) from a fixed Whittaker functional $\lambda ^{\vee }$ for $\rho ^{\vee }$ . By the uniqueness of Whittaker functionals, $\Lambda _{\chi } \circ \varphi = c\cdot \Lambda _{\chi }^{\vee }$ for some constant c. Furthermore, since $\varphi $ is induced from an isomorphism $\varphi : \rho ^{\vee } \mapsto \rho $ , it follows immediately that c does not depend on $\chi $ . Therefore, we have

$$\begin{align*}\Lambda_{\chi} \circ \text{sp}_{\chi} \circ T = c\cdot \Lambda_{\chi}^{\vee} \circ J(\chi^{-1}) \circ \text{sp}_{\chi^{-1}}. \end{align*}$$

Note that there is a natural way to normalize $\varphi $ in such a way that $c = 1$ . We denote by $g^{\tau }$ the transpose of an element $g \in GL_k(F)$ with respect to the antidiagonal (and with $g^{-\tau }$ its inverse). One can then define a new representation $\rho _1$ by $\rho _1(g) = \rho (g^{-\tau })$ . This representation is isomorphic to the contragredient of $\rho $ ; the advantage is that it acts on $V_{\rho }$ , the space of $\rho $ . Furthermore, for any diagonal matrix (i.e., an element of the maximal torus) $t \in GL_k(F)$ , we may conjugate $\rho _1$ to get $\rho _2(g) = {}^t\rho _1(g) = \rho _1(t^{-1}gt)$ . Then $\rho _2 \cong \rho _1$ , and with a suitable choice of t, $\rho _2$ becomes $\psi $ -generic with the same Whittaker functional $\lambda $ . For example, assume that $\psi $ is given by

$$\begin{align*}\psi(u) = \psi_0(u_{1,2}+\dotsb+u_{k-1,k}), \end{align*}$$

where $\psi _0$ is a nontrivial additive character of F, and u is an upper-triangular unipotent matrix with entries $u_{1,2},\dotsc ,u_{k-1,k}$ above the main diagonal. Then one checks immediately that $t=\text {diag}(1,-1,\dotsc ,(-1)^{k-1})$ gives

$$\begin{align*}\lambda(\rho_2(u)v) = \psi(u)\lambda(v) \end{align*}$$

for any $v \in V_{\rho }$ . In short, we may assume that $ \Lambda _{\chi } \circ \text {sp}_{\chi } \circ T = \Lambda _{\chi }^{\vee } \circ J(\chi ^{-1}) \circ \text {sp}_{\chi ^{-1}}$ .

This leads to the second choice we have to make in the construction of T: that of the normalization of the intertwining operator J. Here, we choose the standard normalization introduced by Shahidi (cf. [Reference Shahidi24, Theorem 3.1]). Under this assumption, we have

$$\begin{align*}\Lambda_{\chi}^{\vee} \circ J(\chi^{-1}) = \Lambda_{\chi^{-1}} \end{align*}$$

for every unramified character $\chi $ . Thus,

$$\begin{align*}\Lambda_{\chi} \circ \text{sp}_{\chi} \circ T = \Lambda_{\chi^{-1}} \circ \text{sp}_{\chi^{-1}}. \end{align*}$$

With this, we are ready to compare the action of T on $\pi $ and $\pi ^-$ . For $\pi $ , we specialize at $\chi = 1$ ; this gives us

$$\begin{align*}\Lambda_1 \circ \text{sp}_1 \circ T =\Lambda_1 \circ \text{sp}_1, \end{align*}$$

i.e., T acts trivially.

For $\pi ^-$ , we specialize at $\chi _0$ . We notice that $\chi _0^{-1} = \chi _0\eta $ for some character $\eta $ such that $\eta \circ \rho \cong \rho $ . This shows that $\text {sp}_{\chi ^{-1}} = \phi _{\eta }\circ \text {sp}_{\chi _0}$ , where $\phi _{\eta }$ is the isomorphism $\rho \mapsto \eta \otimes \rho $ defined in [Reference Heiermann17, Section 1.17] (again, we induce to $\phi _{\eta } : i_P^G(\rho \otimes \tau ) \to i_P^G(\eta \rho \otimes \tau )$ and abuse the notation). Finally, using the uniqueness of Whittaker functionals again, we see that $\Lambda _{\chi \eta } \circ \phi _{\eta } = d\cdot \Lambda _{\chi }$ for some constant d which does not depend on $\chi $ . We can normalize $\phi _{\eta }$ so that $d=1$ ; then we have

$$\begin{align*}\Lambda_{\chi_0} \circ \text{sp}_{\chi_0} \circ T = \Lambda_{\chi_0^{-1}} \circ \text{sp}_{\chi_0^{-1}} = \Lambda_{\chi_0\eta} \circ \phi_{\eta}\circ \text{sp}_{\chi_0}= \Lambda_{\chi_0} \circ \text{sp}_{\chi_0}. \end{align*}$$

Therefore, T acts trivially on $\pi ^-$ as well.

To summarize, if we use Shahidi’s normalization of the standard intertwining operator, and normalize $\varphi $ as we did above, it follows that T acts trivially on both $\pi $ and $\pi ^-$ . This implies that the Gelfand–Graev module is isomorphic to

$$\begin{align*}\mathcal{H} \otimes_{\mathcal{H}_0} \epsilon_0 \end{align*}$$

(see (3.10)), where $\epsilon _0(T_n) = 1$ . Note that this is analogous to our results in Case (iii), because $T_n$ again acts by $q^r$ , only this time $r= 0$ .

This completes our analysis of the structure of $\mathcal {H}$ . We conclude the section by providing an alternative proof for the following result of [Reference Bushnell and Henniart6].

Corollary 3.8 We have

$$\begin{align*}{\mathrm{End}}_{\mathcal{H}}(\Pi) \cong Z(\mathcal{H}), \end{align*}$$

the center of $\mathcal {H}$ .

Proof Obviously, $Z(\mathcal {H})$ is contained in ${\mathrm {End}}_{\mathcal {H}}(\Pi )$ , so we need to prove that any element of ${\mathrm {End}}_{\mathcal {H}}(\Pi )$ is given by a multiplication with an element $f \in Z(\mathcal {H})$ . We prove the corollary in Case (iii); the proof in Cases (i) and (ii) is analogous.

We start by recalling that $Z(\mathcal {H}) = \mathcal {A}^W$ , the Weyl group invariants of $\mathcal {A}$ . Note that any element of ${\mathrm {End}}_{\mathcal {H}}(\Pi )$ can be viewed as an element of $\mathcal {A}$ . Indeed, let $f \in {\mathrm {End}}_{\mathcal {H}}(\Pi )$ . We have ${\mathrm {End}}_{\mathcal {H}}(\Pi ) \subseteq {\mathrm {End}}_{\mathcal {A}}(\Pi )$ , but we know that $\Pi = \mathcal {A}$ as an $\mathcal {A}$ -module. Therefore, $f \in {\mathrm {End}}_{\mathcal {A}}(\mathcal {A}) = \mathcal {A}$ . Thus, it remains to prove that f is invariant under the action of the Weyl group.

It suffices to prove that f is invariant under the set of simple reflections which generate the Weyl group. In other words, we need to prove that

$$\begin{align*}f^{\vee} = f \quad \text{and}\quad f^{s_i} = f, \quad i= 1,\dotsc,n-1, \end{align*}$$

using the notation of Section 2.6. This follows immediately from what we now know about the structure of $\Pi $ as an $\mathcal {H}$ -module: $\Pi = \mathcal {H} \otimes _{\mathcal {H}_0} \epsilon $ . In other words, we have shown that there exists an element $g\in \mathcal {A} \cong \Pi $ (constructed in Section 3.1) on which the elements $T_1,\dotsc , T_{n-1}$ and $T_n$ act by scalar multiplication with $q^t$ , and $q^r$ , respectively.

We now look at the commutation relation

$$\begin{align*}T_nf - f^{\vee} T_n = \left((q^r-1) + \frac{1}{X_n}(\sqrt{q}^{r+s}-\sqrt{q}^{r-s})\right)\frac{f - f^{\vee}}{1-1/X_n^2} \end{align*}$$

satisfied by $T_n$ and f. Applying this to g (recall that $T_ng = q^rg$ ), and using the fact that f is in ${\mathrm {Hom}}_{\mathcal {H}}(\Pi )$ (so that $T_nfg = fT_ng$ ), we get

$$\begin{align*}(f-f^{\vee})q^r\cdot g = \left((q^r-1) + \frac{1}{X_n}(\sqrt{q}^{r+s}-\sqrt{q}^{r-s})\right)\frac{f - f^{\vee}}{1-1/X_n^2}\cdot g. \end{align*}$$

This is an equality in $\mathcal {A}$ . Since $q^r \neq \left ((q^r-1) + \frac {1}{X_n}(\sqrt {q}^{r+s}-\sqrt {q}^{r-s})\right )\frac {1}{1-1/X_n^2}$ and ${g\neq 0}$ , it follows that $f-f^{\vee }$ must be $0$ . Therefore, $f = f^{\vee }$ . We get $f = f^{s_i}$ in the same way, using the commutation relations satisfied by the operators $T_i$ . This proves the corollary.