2.1 Unitary groups

Let $V = V_{2n+1}$ (resp. $W = W_{2n}$ ) be a hermitian (resp. skew-hermitian) space over E of dimension $2n+1$ (resp. $2n$ ) equipped with a nondegenerate hermitian form $\left \langle \cdot , \cdot \right \rangle _V$ (resp. skew-hermitian form $\left \langle \cdot ,\cdot \right \rangle _W$ ). Assume that there are bases $\{e_n, \dots , e_1, e_0, e_{-1}, \dots , e_{-n}\}$ of V and $\{f_n,\dots ,f_1, f_{-1}, \dots , f_{-n}\}$ of W, respectively, such that

$$\begin{align*}\left\langle e_i, e_j \right\rangle_V = \left\langle f_i, f_j \right\rangle_W = 0 \end{align*}$$

unless $j = -i$ , and

$$\begin{align*}\left\langle e_0,e_0 \right\rangle_V = \left\langle e_i,e_{-i} \right\rangle_V = \left\langle f_i,f_{-i} \right\rangle_W = 1 \end{align*}$$

for $1 \leq i \leq n$ .

Using these bases, we often identify the associated unitary groups $\mathrm {U}(V)$ and $\mathrm {U}(W)$ with

$$ \begin{align*} \mathrm{U}_{2n+1} &= \left\{ h \in \mathrm{GL}_{2n+1}(E) \;\middle|\; {}^t\overline{h} w_{2n+1} h = w_{2n+1} \right\}, \\ \mathrm{U}_{2n} &= \left\{ g \in \mathrm{GL}_{2n}(E) \;\middle|\; {}^t\overline{g} J_{2n} g = J_{2n} \right\}, \end{align*} $$

respectively, where we set

$$\begin{align*}w_n = \begin{pmatrix} &&1\\ &\unicode{x22F0}&\\ 1&& \end{pmatrix} \in \mathrm{GL}_n(E), \quad J_{2n} = \begin{pmatrix} 0&w_n\\ -w_n&0 \end{pmatrix} \in \mathrm{GL}_{2n}(E). \end{align*}$$

2.2 Representations of unitary groups

Let $N_{2n+1}$ (resp. $N_{2n}$ ) be the upper triangular unipotent subgroup of $\mathrm {U}_{2n+1}$ (resp. $\mathrm {U}_{2n}$ ). We define generic characters of $N_{2n+1}$ and $N_{2n}$ by the same formula

$$\begin{align*}u \mapsto \psi_E\left(\sum_{k=1}^{n}u_{k,k+1}\right). \end{align*}$$

By abuse of notation, we denote these characters by $\psi _E$ . We say that an irreducible representation $\sigma $ of $\mathrm {U}_{2n+1}$ (resp. $\pi $ of $\mathrm {U}_{2n}$ ) is generic (resp. $\psi _E$ -generic) if $\mathrm {Hom}_{N_{2n+1}}(\sigma , \psi _E) \not = 0$ (resp. $\mathrm {Hom}_{N_{2n}}(\pi , \psi _E) \not = 0$ ).

For an irreducible representation $\pi $ of $\mathrm {U}_{2n}$ , we denote by $\pi ^\vee $ the contragredient representation of $\pi $ . By a result in [Reference Mœglin, Vignéras and Waldspurger19, Chapter 4. II. 1], we know $\pi ^\vee \cong \pi ^\theta $ , where $\pi ^\theta (g) = \pi (\theta (g))$ with

$$\begin{align*}\theta \colon \mathrm{U}_{2n} \rightarrow \mathrm{U}_{2n},\; g \mapsto \begin{pmatrix} \mathbf{1}_n & 0 \\ 0 & -\mathbf{1}_n \end{pmatrix} \overline{g} \begin{pmatrix} \mathbf{1}_n & 0 \\ 0 & -\mathbf{1}_n \end{pmatrix}^{-1}. \end{align*}$$

In particular, $\pi $ is $\psi _E$ -generic if and only if $\pi ^\vee $ is $\psi _E^{-1}$ -generic.

By the local Langlands correspondence established by Mok [Reference Mok20], to an irreducible representation $\sigma $ of $\mathrm {U}_{2n+1}$ (resp. $\pi $ of $\mathrm {U}_{2n}$ ), one can attach a conjugate self-dual representation $\phi _\sigma $ (resp. $\phi _\pi $ ) of $W_E \times \mathrm {SL}_2(\mathbb {C})$ of dimension $2n+1$ (resp. $2n$ ), where $W_E$ is the Weil group of E. We call $\phi _\sigma $ (resp. $\phi _\pi $ ) the L-parameter for $\sigma $ (resp. $\pi $ ). Then we define the conductor $c(\phi _\sigma )$ of $\phi _\sigma $ by the non-negative integer satisfying

$$\begin{align*}\varepsilon(s,\phi_\sigma,\psi_E) = \varepsilon(0,\phi_\sigma,\psi_E)q^{-2c(\phi_\sigma)s}. \end{align*}$$

Similarly, the conductor $c(\phi _\pi )$ of $\phi _\pi $ is defined.

The center of $\mathrm {U}_{2n+1}$ (resp. $\mathrm {U}_{2n}$ ) is $\mathrm {U}_1$ which is identified with $E^1$ . For an irreducible representation $\sigma $ (resp. $\pi $ ) of $\mathrm {U}_{2n+1}$ (resp. $\mathrm {U}_{2n}$ ), we denote its central character by $\omega _\sigma $ (resp. $\omega _\pi $ ). If $\sigma $ (resp. $\pi $ ) corresponds to $\phi _\sigma $ (resp. $\phi _\pi $ ), then the L-parameter of $\omega _\sigma $ (resp. $\omega _\pi $ ) is given by $\det (\phi _\sigma )$ (resp. $\det (\phi _\pi )$ ).

2.3 Jacobi group

Set

$$\begin{align*}\mathbf{v}(x,y;z) = \begin{pmatrix} 1 & x & y & z+\frac{1}{2}(x w_{n-1}{}^t\overline{y} - y w_{n-1} {}^t\overline{x})\\ 0 & \mathbf{1}_{n-1} & 0 & w_{n-1}{}^t\overline{y} \\ 0 & 0& \mathbf{1}_{n-1} & -w_{n-1}{}^t\overline{x} \\ 0&0&0& 1 \end{pmatrix} \in \mathrm{U}_{2n} \end{align*}$$

for $x,y \in E^{n-1}$ and $z \in F$ . Here, $E^{n-1}$ is the space of row vectors. Let $H_{n-1} = \{\mathbf {v}(x,y; z) \;|\; x,y \in E^{n-1}, z \in F\} \cong E^{2n-2} \oplus F$ be a Heisenberg group in $4n-3$ variables over F with the multiplication law

$$\begin{align*}\mathbf{v}(x,y;z) \mathbf{v}(x',y';z') = \mathbf{v}\left(x+x', y+y'; z+z'+\frac{1}{2}\mathrm{tr}_{E/F}(x w_{n-1}{}^t\overline{y} - y w_{n-1} {}^t\overline{x}) \right). \end{align*}$$

We write

$$ \begin{align*} X_{n-1} &= \{\mathbf{v}(x,0;0) \;|\; x \in E^{n-1}\}, \\ Y_{n-1} &= \{\mathbf{v}(0,y;0) \;|\; y \in E^{n-1}\}, \\ Z &= \{\mathbf{v}(0,0;z) \;|\; z \in F\}. \end{align*} $$

By abuse of notation, we denote the character $Z \ni \mathbf {v}(0,0;z) \mapsto \psi (z)$ by $\psi $ .

We identify $\mathrm {U}_{2n-2}$ as a subgroup of $\mathrm {U}_{2n}$ by the inclusion

$$\begin{align*}\mathrm{U}_{2n-2} \ni g' \mapsto \begin{pmatrix} 1 && \\ &g'& \\ &&1 \end{pmatrix} \in \mathrm{U}_{2n}. \end{align*}$$

Then $\mathrm {U}_{2n-2}$ normalizes $H_{n-1}$ . We call $J_{n-1} = H_{n-1} \rtimes \mathrm {U}_{2n-2}$ the Jacobi group. Note that Z is the center of $J_{n-1}$ .

For an irreducible representation $\pi $ of $\mathrm {U}_{2n}$ , we denote by $\pi _{\psi }$ the maximal quotient of $\pi $ on which Z acts by $\psi $ . We call $\pi _\psi $ the Fourier–Jacobi module of $\pi $ . For a compact open subgroup K of $\mathrm {U}_{2n}$ , we denote by $\pi ^K_{\psi }$ the image of $\pi ^K$ via the canonical surjection $\pi \twoheadrightarrow \pi _{\psi }$ . Note that $\pi _{\psi }$ is a smooth representation of $J_{n-1}$ so that K does not act on $\pi _{\psi }$ itself.

For $t \in E^\times $ , if we put $\psi '(x) = \psi (N_{E/F}(t)x)$ and

$$\begin{align*}K' = \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ && \overline{t}^{-1} \end{pmatrix}^{-1} K \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ && \overline{t}^{-1} \end{pmatrix}, \end{align*}$$

then $\pi (\mathrm {diag}(t,\mathbf {1}_{2n-2}, \overline {t}^{-1}))$ induces isomorphisms

$$\begin{align*}\pi^{K'} \xrightarrow{\sim} \pi^K,\quad \pi_{\psi'} \xrightarrow{\sim} \pi_\psi. \end{align*}$$

Hence, we have $\pi _{\psi '}^{K'} \cong \pi _{\psi }^K$ .

2.4 Compact subgroups

For each non-negative even integer $2m \geq 0$ , we define compact subgroups $K_{2m}^V \subset \mathrm {U}(V) \cong \mathrm {U}_{2n+1}$ and $K_{2m}^W \subset \mathrm {U}(W) \cong \mathrm {U}_{2n}$ as follows. When $2m = 0$ , we set $K_0^V = \mathrm {U}_{2n+1} \cap \mathrm {GL}_{2n+1}(\mathfrak {o}_E)$ and $K_0^W = \mathrm {U}_{2n} \cap \mathrm {GL}_{2n}(\mathfrak {o}_E)$ . If $2m> 0$ , we set

Note that

which is denoted by $\mathbb {K}_{2m,\mathrm {U}(V)}$ in [Reference Atobe, Oi and Yasuda3], and by $K_{n,2m}$ in [Reference Cheng8]. If we set ${}^tK_{2m}^W = \{{}^tk \;|\; k \in K_{2m}^W\}$ to be the transpose of $K_{2m}^W$ , then

$$ \begin{align*} K_{2m}^W = \begin{pmatrix} \varpi^{-m} && \\ &\mathbf{1}_{2n-2}& \\ && \varpi^m \end{pmatrix} {}^tK_{2m}^W \begin{pmatrix} \varpi^{-m} && \\ &\mathbf{1}_{2n-2}& \\ && \varpi^m \end{pmatrix}^{-1}. \end{align*} $$

The theory of local newforms for $\mathrm {U}_{2n+1}$ is established by the author together with Oi and Yasuda [Reference Atobe, Oi and Yasuda3, Theorem 1.1] and by Cheng [Reference Cheng8, Theorem 1.2] as follows.

In this paper, we will prove an analogue of this theorem for $\mathrm {U}_{2n}$ as follows.

When $2m = c(\phi _\pi )$ , we shall call an element in $\pi ^{K_{2m}^W}$ whose image in $\pi _\psi $ is nonzero a local newform of $\pi $ .

3.1 Weil representation

Let $W_0$ be the subspace of W generated by $\{f_{n-1},\dots ,f_1, f_{-1}, \dots , f_{-n+1}\}$ . We write $G_n = \mathrm {U}(W)$ and $G_{n-1} = \mathrm {U}(W_0)$ in this section. Hence, the Jacobi group $J_{n-1}$ is written as $J_{n-1} = H_{n-1} \rtimes G_{n-1}$ .

Recall that we have a compact subgroup $K_{2m}^W$ of $G_n = \mathrm {U}(W)$ . Note that the intersections

$$\begin{align*}K^J = K_{2m}^{W} \cap J_{n-1}, \quad K^H = K_{2m}^{W} \cap H_{n-1}, \quad K^{W_0} = K_{2m}^{W} \cap \mathrm{U}(W_0) \end{align*}$$

are independent of $2m$ . Moreover, $K^{W_0}$ is a hyperspecial maximal compact subgroup of $G_{n-1} = \mathrm {U}(W_0)$ .

We consider the Weil representation $\omega _\psi $ of $J_{n-1}$ associated to $\psi $ and $\chi $ . It is realized on the Schwartz space $\mathcal {S}(E^{n-1})$ as follows. For $\phi \in \mathcal {S}(E^{n-1})$ and $\xi \in E^{n-1}$ ,

$$ \begin{align*} &\omega_\psi(\mathbf{v}(x,0;0))\phi(\xi) = \phi(\xi+x), \quad x \in E^{n-1}, \\ &\omega_\psi(\mathbf{v}(0,y;0))\phi(\xi) = \psi_E(2\xi w_{n-1} {}^t\overline{y})\phi(\xi), \quad y \in E^{n-1}, \\ &\omega_\psi(\mathbf{v}(0,0;z))\phi(\xi) = \psi(z) \phi(\xi), \quad z \in F, \\ &\omega_\psi(\mathbf{m}(a))\phi(\xi) = \chi(\det(a)) |\det(a)|^{\frac{1}{2}} \phi(\xi a), \quad a \in \mathrm{GL}_{n-1}(E), \\ &\omega_\psi(\mathbf{n}(b))\phi(\xi) = \psi_E\left(\overline{\xi} \overline{b} w_{n-1} {}^t \xi \right)\phi(\xi), \quad b \in \mathrm{M}_{n-1}(E), {}^t(w_{n-1}\overline{b}) = w_{n-1}b, \\ &\omega_\psi (J_{2n-2})\phi(\xi) = \int_{E^{n-1}} \phi(x) \psi_E(2\overline{x} \cdot {}^t\xi) dx, \end{align*} $$

where we set

$$\begin{align*}\mathbf{m}(a) = \begin{pmatrix} a & 0 \\ 0 & w_{n-1}{}^t\overline{a}^{-1}w_{n-1}^{-1} \end{pmatrix}, \quad \mathbf{n}(b) = \begin{pmatrix} 1 & b \\ 0 & 1 \end{pmatrix} \in G_{n-1}, \end{align*}$$

and the measure $dx$ on $E^{n-1}$ is the self-dual Haar measure with respect to $\psi _E$ . The Weil representation $\omega _\psi $ is unitary with respect to the pairing

$$\begin{align*}(\phi_1,\phi_2) = \int_{E^{n-1}}\phi_1(\xi) \overline{\phi_2(\xi)} d\xi. \end{align*}$$

Set $\phi _0 \in \mathcal {S}(E^{n-1})$ to be the characteristic function on $\mathfrak {o}_E^{n-1}$ . Note that $\phi_0 $ is fixed by $\omega _{\psi }(K^J)$ . Moreover, the subspace $\omega _{\psi }^{K^H}$ is one-dimensional spanned by $\phi _0$ .

3.2 Proof of Theorem 2.2 (1)

Let $\pi \in \mathrm {Irr}_{\mathrm {temp}}(G_n)$ and $\pi ' \in \mathrm {Irr}_{\mathrm {temp}}(G_{n-1})$ . Fix a nonzero $G_n$ -invariant (resp. $G_{n-1}$ -invariant) bilinear pairing $(\cdot , \cdot )_{\pi } \colon \pi \times \pi ^\vee \rightarrow \mathbb {C}$ (resp. $(\cdot , \cdot )_{\pi '} \colon \pi ' \times \pi ^{\prime \vee } \rightarrow \mathbb {C}$ ). For $\varphi \in \pi , \varphi ^\vee \in \pi ^\vee , \varphi ' \in \pi ', \varphi ^{\prime \vee } \in \pi ^{\prime \vee }$ and $\phi , \phi ^\vee \in \mathcal {S}(E^{n-1})$ , we define the local Fourier–Jacobi period by

$$ \begin{align*} &\alpha(\varphi, \varphi^\vee, \varphi', \varphi^{\prime\vee}, \phi, \phi^\vee) \\&= \int_{G_{n-1}}\int_{H_{n-1}} (\pi(hg)\varphi, \varphi^\vee)_{\pi} (\pi'(g)\varphi', \varphi^{\prime\vee})_{\pi'} \overline{(\omega_\psi(hg) \phi, \phi^\vee)} dhdg. \end{align*} $$

Since the central character of $\omega _\psi $ is $\psi $ , if $\alpha (\varphi , \varphi ^\vee , \varphi ', \varphi ^{\prime \vee }, \phi , \phi ^\vee ) \not = 0$ , then

$$\begin{align*}\int_{F}(\pi(hg \cdot \mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \not= 0 \end{align*}$$

for some $h \in H_{n-1}$ and $g \in G_{n-1}$ . This means that the image of $\varphi $ in $\pi _\psi $ is nonzero. The converse holds in the following sense.

Lemma 3.2. Let $\varphi \in \pi $ . Assume that the image of $\varphi $ in $\pi _\psi $ is nonzero. Then there exists $\varphi ^\vee \in \pi ^\vee $ such that

$$\begin{align*}\int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \not= 0. \end{align*}$$

Proof. Note that by Proposition 3.1, the integral

$$\begin{align*}\int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \end{align*}$$

converges absolutely. Suppose that this integral is equal to zero for all $\varphi ^\vee \in \pi ^\vee $ . We will show that the image of $\varphi $ in $\pi _\psi $ is zero.

For an integer $j> 0$ , set

$$\begin{align*}T_j = \left\{ t(1+a) = \begin{pmatrix} 1+a && \\ &\mathbf{1}_{2n-2}& \\ && (1+a)^{-1} \end{pmatrix} \;\middle|\; a \in \mathfrak{p}_F^j \right\} \subset \mathrm{U}_{2n}. \end{align*}$$

Recall that $\psi |_{\mathfrak {o}_F} = \mathbf {1}$ but $\psi |_{\mathfrak {p}_F^{-1}} \not = \mathbf {1}$ . Hence, for fixed $z \in F$ with $-k = \mathrm {ord}(z)$ , the map

$$\begin{align*}T_j \ni t(1+a) \mapsto \frac{\psi((1+a)^2z)}{\psi(z)} \in \mathbb{C}^\times \end{align*}$$

is a character if $k \leq 2j$ . Moreover, it is trivial if $k \leq j$ . Hence,

$$\begin{align*}q^{-j} \int_{\mathfrak{p}_F^{j}} \psi((1+a)^2z) da = \left\{ \begin{aligned} &\psi(z) &\quad&\text{if } k \leq j, \\ &0 &\quad&\text{if } j < k \leq 2j. \end{aligned} \right. \end{align*}$$

Since $\pi $ is smooth, there is an integer $j> 0$ such that $\varphi $ is $T_j$ -fixed. To show that the image of $\varphi $ in $\pi _\psi $ is zero, it suffices to prove that

$$\begin{align*}\int_{\mathfrak{p}_F^{-j}} \pi(\mathbf{v}(0,0;z)\varphi) \overline{\psi(z)} dz = 0. \end{align*}$$

This is equivalent to saying that

(†) $$ \begin{align} \int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz = 0 \end{align} $$

for all $\varphi ^\vee \in \pi ^\vee $ . We claim that we may assume that $\varphi ^\vee $ is $T_j$ -fixed. Indeed, if $z \in \mathfrak {p}_F^{-j}$ , since $k = -\mathrm {ord}(z) \leq j$ , we have

$$ \begin{align*} \int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz &= \int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \left(q^{-j}\int_{\mathfrak{p}_F^j}\overline{\psi((1+a)^2z)} da\right)dz \\&= q^{-j} \int_{\mathfrak{p}_F^{-j}} \int_{\mathfrak{p}_F^j} (\pi(t(1+a)^{-1}\mathbf{v}(0,0;z)t(1+a)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} da dz \\&= q^{-j} \int_{\mathfrak{p}_F^{-j}} \int_{\mathfrak{p}_F^j} (\pi(\mathbf{v}(0,0;z)) \varphi, \pi^\vee(t(1+a))\varphi^\vee)_\pi \overline{\psi(z)} da dz. \end{align*} $$

Hence, $(\dagger )$ holds for $\varphi ^\vee $ if it holds for

$$\begin{align*}q^{-j} \int_{\mathfrak{p}_F^j} \pi^\vee(t(1+a))\varphi^\vee da \end{align*}$$

which is $T_j$ -fixed.

Now assume that $\varphi ^\vee $ is $T_j$ -fixed. Then we claim that

$$\begin{align*}\int_{\mathfrak{p}_F^{-j}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz = \int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz, \end{align*}$$

and hence, the left-hand side is zero by assumption. Indeed, for $k> j > 0$ , since $k \geq 2$ so that $k-1 < k \leq 2(k-1)$ , we have

$$ \begin{align*} &\int_{\mathfrak{p}_F^{-k} \setminus \mathfrak{p}_F^{-k+1}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \\&= q^{-k+1} \int_{\mathfrak{p}_F^{-k} \setminus \mathfrak{p}_F^{-k+1}} \int_{\mathfrak{p}_F^{k-1}} (\pi(t(1+a)^{-1}\mathbf{v}(0,0;z)t(1+a))\varphi, \varphi^\vee)_\pi \overline{\psi(z)} da dz \\&= \int_{\mathfrak{p}_F^{-k} \setminus \mathfrak{p}_F^{-k+1}}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \left(q^{-k+1}\int_{\mathfrak{p}_F^{k-1}} \overline{\psi((1+a)^2z)} da \right)dz \\&=0. \end{align*} $$

This completes the proof of the lemma.

Now, we prove Theorem 2.2 (1).

Proof of Theorem 2.2 (1).

Let $\pi $ be an irreducible tempered representation of $G_n = \mathrm {U}_{2n}$ . Suppose that $\pi _{\psi }^{K_{2m}^W} \not = 0$ for some $2m \geq 0$ . We will show that $\pi $ must be $\psi _E$ -generic.

Fix $\varphi \in \pi ^{K_{2m}^W}$ such that the image of $\pi _{\psi }$ is nonzero. By Lemma 3.2, one can find $\varphi ^\vee \in \pi ^\vee $ such that

$$\begin{align*}\int_{F}(\pi(\mathbf{v}(0,0;z)) \varphi, \varphi^\vee)_\pi \overline{\psi(z)} dz \not= 0. \end{align*}$$

Since Z is the center of $H_{n-1}$ , we may assume that $\varphi ^\vee $ is fixed by $K^H$ . Hence, the matrix coefficient $H_{n-1} \ni h \mapsto (\pi (h) \varphi , \varphi ^\vee )_\pi $ is bi- $K^H$ -invariant. Since $\omega _{\psi }$ is the unique irreducible representation of $H_{n-1}$ whose central character is $\psi $ , there are $\phi , \phi ^\vee \in \mathcal {S}(E^{n-1})$ such that

$$\begin{align*}\int_{H_{n-1}} (\pi(h) \varphi, \varphi^\vee)_\pi \overline{(\omega_{\psi}(h)\phi, \phi^\vee)} dh \not= 0. \end{align*}$$

We may also assume that both $\phi $ and $\phi ^\vee $ are fixed by $K^H$ . Since $\omega _{\psi }^{K^H} = \mathbb {C} \phi _0$ , we can take $\phi = \phi ^\vee = \phi _0$ . Hence,

$$\begin{align*}\int_{H_{n-1}} (\pi(h) \varphi, \varphi^\vee)_\pi \overline{(\omega_{\psi}(h)\phi_0, \phi_0)} dh \not= 0. \end{align*}$$

Now by applying the same argument as [Reference Gan and Savin11, Lemma 12.5] to the integral on $G_{n-1}$ , one can find $\pi ' \in \mathrm {Irr}_{\mathrm {temp}}(G_{n-1})$ and $(\varphi ', \varphi ^{\prime \vee }) \in \pi ' \times \pi ^{\prime \vee }$ such that

$$\begin{align*}\alpha(\varphi, \varphi^\vee, \varphi', \varphi^{\prime\vee}, \phi_0, \phi_0) \not= 0. \end{align*}$$

We may assume that $\varphi '$ is fixed by $K^{W_0}$ since so are $\varphi $ and $\phi _0$ . This means that $\pi '$ is unramified. By the local Gan–Gross–Prasad conjecture ([Reference Gan, Gross and Prasad9, Conjecture 17.3, Theorem 19.1]), whose basic case is proven by Gan–Ichino [Reference Gan and Ichino10, Theorem 1.3], we can deduce that $\pi $ is $\psi _E$ -generic.

Conversely, if $\pi $ is $\psi _E$ -generic, by the local Gan–Gross–Prasad conjecture, one can find an irreducible tempered unramified representation $\pi '$ of $G_{n-1}$ such that $\mathrm {Hom}_{J_{n-1}}(\pi \otimes \pi ' \otimes \overline {\omega _{\psi }}, \mathbb {C}) \not = 0$ . Since $\pi '$ and $\omega _\psi $ are irreducible as representations of $G_{n-1}$ and $H_{n-1}$ , respectively, for any nonzero unramified vector $\varphi ^{\prime }_0 \in \pi '$ and for any nonzero element $\mathcal {L} \in \mathrm {Hom}_{J_{n-1}}(\pi \otimes \pi ' \otimes \overline {\omega _{\psi }}, \mathbb {C})$ , one can take $\varphi \in \pi $ such that $\mathcal {L}(\varphi \otimes \varphi _0' \otimes \overline {\phi _0}) \not = 0$ . We may assume that $\varphi $ is fixed by $K^{J}$ . Since $\pi $ is smooth, $\varphi $ is fixed by $K_{2m}^{W}$ for $2m \gg 0$ . In this case, $\varphi $ gives a nonzero element in $\pi _\psi ^{K_{2m}^W}$ .

This completes the proof of Theorem 2.2 (1).

Recall in [Reference Gan, Gross and Prasad9, Corollary 16.3] that for $\pi \in \mathrm {Irr}(G_n)$ and $\pi ' \in \mathrm {Irr}(G_{n-1})$ , we have

$$\begin{align*}\dim_{\mathbb{C}} \mathrm{Hom}_{J_{n-1}}(\pi \otimes \pi' \otimes \overline{\omega_\psi}, \mathbb{C}) \leq 1. \end{align*}$$

It is worth to state the following result which was obtained by the above argument.

4.1 Rankin–Selberg integrals

Let $\tau $ be an irreducible generic representation of $\mathrm {GL}_{n-1}(E)$ which is realized on the Whittaker space $\mathcal {W}(\tau ,\psi _E^{-1})$ with respect to the inverse of $\psi _E$ . For $s \in \mathbb {C}$ , we consider the normalized parabolically induced representation

$$\begin{align*}\mathrm{Ind}_{Q_{n-1}}^{G_{n-1}}\left(\tau |\det|^{s-\frac{1}{2}}\right) \end{align*}$$

of $G_{n-1}$ , where $Q_{n-1} = M_{n-1}U_{n-1}$ denotes the standard Siegel parabolic subgroup so that

$$ \begin{align*} M_{n-1} &= \{\mathbf{m}(a) \;|\; a \in \mathrm{GL}_{n-1}(E)\}, \\ U_{n-1} &= \{\mathbf{n}(b) \;|\; b \in \mathrm{M}_{n-1}(E), {}^t(w_{n-1}\overline{b}) = w_{n-1}b\}. \end{align*} $$

We realize it on the space $V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau ,\psi _E^{-1}), s)$ of smooth functions $f_s \colon G_{n-1} \times \mathrm {GL}_{n-1}(E) \rightarrow \mathbb {C}$ such that

• • $f_s(\mathbf {n}(b) \mathbf {m}(a) g,a') = |\det a|_E^{s+\frac {n}{2}-1} f_s(g,a'a)$ for $g \in G_{n-1}$ , $a,a' \in \mathrm {GL}_{n-1}(E)$ and $\mathbf {n}(b) \in U_{n-1}$ ;

• • the function $a \mapsto f_s(g,a)$ belongs to $\mathcal {W}(\tau ,\psi _E^{-1})$ for any $g \in G_{n-1}$ .

Define a new representation $\tau ^*$ by $\tau ^*(a) = \tau (a^*)$ , where $a^* = w_{n-1}{}^t\overline {a}^{-1}w_{n-1}^{-1}$ . Note that $\tau ^* \cong \overline {\tau }^\vee $ , where $\overline {\tau }(a) = \tau (\overline {a})$ . As in [Reference Morimoto21, Section 2.3], one can define a normalized intertwining operator

$$\begin{align*}M^*(\tau,s) \colon V_{Q_{n-1}}^{G_{n-1}}(\mathcal{W}(\tau,\psi_E^{-1}), s) \rightarrow V_{Q_{n-1}}^{G_{n-1}}(\mathcal{W}(\tau^*,\psi_E^{-1}), 1-s). \end{align*}$$

Let $\pi $ be an irreducible $\psi _E$ -generic representation of $G_n$ realized on the Whittaker space $\mathcal {W}(\pi ,\psi _E)$ . For $W \in \mathcal {W}(\pi ,\psi _E)$ , $f_s \in V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau ,\psi _E^{-1}), s)$ and $\phi \in \mathcal {S}(E^{n-1})$ , we define the Rankin–Selberg integral $\mathcal {L}(W,f_s,\overline {\phi })$ by

$$\begin{align*}\int_{N_{n-1} \backslash G_{n-1}} \int_{E^{n-1}} W(w_{1,n-1} \mathbf{v}(x,0;0)g ) f_s(g,\mathbf{1}_{n-1}) \overline{\omega_\psi(g) \phi(x)} dx dg, \end{align*}$$

where we set

$$\begin{align*}w_{1,n-1} = \left(\begin{array}{cc|cc} &\mathbf{1}_{n-1}&&\\ 1&&&\\ \hline &&&1\\ &&\mathbf{1}_{n-1}& \end{array}\right) \in G_n. \end{align*}$$

Remark 4.1. Note that

$$\begin{align*}W(w_{1,n-1} \mathbf{v}(x,0;0)g \cdot \mathbf{v}(0,0;z)) = \psi(z)W(w_{1,n-1}\mathbf{v}(x,0;0)g) \end{align*}$$

for $W \in \mathcal {W}(\pi , \psi _E)$ . Hence, the restriction map $W \mapsto W(w_{1,n-1} \mathbf {v}(x,0;0)g)$ factors through $\pi \twoheadrightarrow \pi _\psi $ . In particular, if $\pi $ is $\psi _E$ -generic, then $\pi _\psi $ is nonzero.

Proof. (1) is [Reference Ben-Artzi and Soudry4, Proposition 6.4]. By [Reference Ben-Artzi and Soudry4, Proposition 6.5], we see that $1 \in I(\pi \times \tau \times \chi )$ , which implies (2). The assertion (3) follows from the multiplicity one theorem proven in [Reference Gan, Gross and Prasad9, Corollary 16.3]. (4) is proven by Morimoto [Reference Morimoto21, Theorem 3.1]. Since $M^*(\tau ^*, 1-s) \circ M^*(\tau , s) = \mathrm {id}$ , using $\omega _{\tau ^*}(-1) = \omega _{\tau }(-1)$ , we have

$$ \begin{align*} \mathcal{L}(W, f_s, \overline{\phi}) &= \mathcal{L}(W, M^*(\tau^*, 1-s) \circ M^*(\tau, s)f_s, \overline{\phi}) \\&= \omega_\pi(-1)^{n-1}\omega_{\tau^*}(-1)^n \Gamma(1-s, \pi \times \tau^*, \chi, \psi) \mathcal{L}(W, M^*(\tau, s)f_s, \overline{\phi}) \\&= \Gamma(1-s, \pi \times \tau^*, \chi, \psi)\Gamma(s, \pi \times \tau, \chi, \psi) \mathcal{L}(W, f_s, \overline{\phi}) \end{align*} $$

for any W, $f_s$ and $\phi $ . It means that

$$\begin{align*}\Gamma(1-s, \pi \times \tau^*, \chi, \psi)\Gamma(s, \pi \times \tau, \chi, \psi) = 1, \end{align*}$$

which is equivalent to saying that

$$\begin{align*}\varepsilon(1-s, \pi \times \tau^*, \chi, \psi)\varepsilon(s, \pi \times \tau, \chi, \psi) = 1. \end{align*}$$

Hence, $\varepsilon (s,\pi \times \tau , \chi , \psi ) \in \mathbb {C}[q^{-s}, q^{s}]^\times = \mathbb {C}^\times (q^{-s})^{\mathbb {Z}}$ .

4.2 Unramified representations

In this subsection, we consider the Rankin–Selberg integrals when $\tau $ varies over irreducible unramified representations of $\mathrm {GL}_{n-1}(E)$ .

Recall that $K^{W_0} = K_{0}^W \cap G_{n-1}$ . It is a hyperspecial maximal compact subgroup of $G_{n-1}$ , and the Iwasawa decomposition $G_{n-1} = Q_{n-1} K^{W_0}$ holds.

Irreducible unramified representations of $\mathrm {GL}_{n-1}(E)$ are parametrized by the Satake parameters $\underline {x} = (x_1,\dots ,x_{n-1}) \in (\mathbb {C}^\times )^{n-1}/S_{n-1}$ . We write the unramified representation associated to $\underline {x}$ by $\tau _{\underline {x}}$ . Then for almost all $\underline {x}$ , since $\tau _{\underline {x}}$ is generic, there exists a unique function $f_{s}(\underline {x}) \in V_{Q_{n-1}}^{G_{n-1}}(\mathcal {W}(\tau_{\underline{x}} ,\psi _E^{-1}), s)$ such that

• • $f_{s}(gk, a; \underline {x}) = f_{s}(g, a; \underline {x})$ for any $g \in G_{n-1}$ , $k \in K^{W_0}$ and $a \in \mathrm {GL}_{n-1}(E)$ ; and

• • the function $W(a; \underline {x}) = f_{s}(\mathbf {1}_{2(n-1)}, a; \underline {x})$ is right $\mathrm {GL}_{n-1}(\mathfrak {o}_E)$ -invariant with $W(\mathbf {1}_{n-1}; \underline {x}) = 1$ .

Lemma 4.3. For $\underline {x} = (x_1, \dots , x_{n-1})$ , we write $\underline {x}^{-1} = (x_1^{-1}, \dots , x_{n-1}^{-1})$ . Then we have

$$ \begin{align*} &\frac{M^*(\tau_{\underline{x}}, s)f_{s}(\underline{x})} {\prod_{i=1}^{n-1}(1-q^{-s}x_i)\prod_{1 \leq i < j \leq n-1}(1-q^{-2s}x_ix_j)} \\ &\quad = \frac{f_{1-s}(\underline{x}^{-1})} {\prod_{i=1}^{n-1}(1-q^{-(1-s)}x_i^{-1})\prod_{1 \leq i < j \leq n-1}(1-q^{-2(1-s)}x_i^{-1}x_j^{-1})}. \end{align*} $$

Let $\pi $ be an irreducible $\psi _E$ -generic tempered representation of $G_n$ with L-parameter $\phi _\pi $ . Then by the uniqueness of the gamma factor (Theorem 4.2 (4)), we have

$$\begin{align*}\Gamma(s, \pi \times \tau_{\underline{x}}, \chi, \psi) = \prod_{i=1}^{n-1} \varepsilon(s+s_i+s_0, \phi_\pi, \psi_E)\frac{L(1-s-s_i-s_0, \phi_\pi^\vee)}{L(s+s_i+s_0, \phi_\pi)} \end{align*}$$

for almost all $\underline {x} = (x_1,\dots ,x_{n-1})$ , where $s_0, s_1, \dots , s_{n-1} \in \mathbb {C}$ are such that $q^{-2s_0} = -1$ and $x_i = q^{-2s_i}$ for $1 \leq i \leq n-1$ . Since $\phi _\pi $ is tempered, two meromorphic functions $\prod _{i=1}^{n-1}L(1-s-s_i-s_0, \phi _\pi ^\vee )$ and $\prod _{i=1}^{n-1}L(s+s_i+s_0, \phi _\pi )$ have no common pole for almost all $\underline {x}$ . In particular, in this case, we have

$$ \begin{align*} L(s, \pi \times \tau_{\underline{x}}, \chi) &= \prod_{i=1}^{n-1}L(s+s_i+s_0, \phi_\pi), \\ \varepsilon(s, \pi \times \tau_{\underline{x}}, \chi, \psi) &= \prod_{i=1}^{n-1} \varepsilon(s+s_i+s_0, \phi_\pi, \psi_E). \end{align*} $$

If we write $L(s, \phi _\pi ) = P_\pi (q^{-2s})$ and $\varepsilon (s, \phi _\pi , \psi _E) = \varepsilon q^{c(\phi _\pi )(1-2s)}$ , then

$$ \begin{align*} L(s, \pi \times \tau_{\underline{x}}, \chi) &= \prod_{i=1}^{n-1}P_\pi(-x_iq^{-2s}), \\ \varepsilon(s, \pi \times \tau_{\underline{x}}, \chi, \psi) &= \varepsilon^{n-1} (-q^{1-2s})^{c(\phi_\pi)(n-1)} \prod_{i=1}^{n-1}x_i^{c(\phi_\pi)}. \end{align*} $$

4.3 Proof of Theorem 2.2 (2)

The symmetric group $S_{n-1}$ acts on $\mathbb {C}[X_1^{\pm 1},\dots , X_{n-1}^{\pm 1}]$ canonically. Set

$$\begin{align*}\mathcal{T} = \mathbb{C}[X_1^{\pm1},\dots,X_{n-1}^{\pm1}]^{S_{n-1}}. \end{align*}$$

Note that

$$\begin{align*}\mathcal{T} = \mathbb{C}[T_1,\dots,T_{n-2}, T_{n-1}, T_{n-1}^{-1}] \end{align*}$$

with

$$\begin{align*}T_i = \sum_{\sigma \in S_{n-1}} X_{\sigma(1)} \cdots X_{\sigma(i)}. \end{align*}$$

The degree with respect to $T_{n-1}$ gives a $\mathbb {Z}$ -grading on $\mathcal {T}$ ; that is, $\mathcal {T} = \oplus _{d \in \mathbb {Z}} \mathcal {T}_d$ with

$$\begin{align*}\mathcal{T}_d = \mathbb{C}[T_1,\dots,T_{n-2}] T_{n-1}^d. \end{align*}$$

Write $\underline {X} = (X_1,\dots ,X_{n-1})$ and $q^{1-2s}\underline {X} = (q^{1-2s}X_1, \dots , q^{1-2s}X_{n-1})$ . There is a function

$$\begin{align*}W(\underline{X}) \colon \mathrm{GL}_{n-1}(E) \rightarrow \mathcal{T} \end{align*}$$

such that $W(\underline {X})|_{\underline {X} = \underline {x}} = W(\underline {x})$ for almost all $\underline {x} \in (\mathbb {C}^\times )^{n-1}$ . Similarly, we consider the function $f_{s}(\underline {X}) \colon G_{n-1} \times \mathrm {GL}_{n-1}(E) \rightarrow \mathcal {T}$ so that $f_{s}(\underline {X})|_{\underline {X} = \underline {x}} = f_{s}(\underline {x})$ for almost all $\underline {x} \in (\mathbb {C}^\times )^{n-1}$ . In particular, $f_{s}(\mathbf {1}_{2(n-1)}, a; \underline {X}) = W(a; q^{1-2s}\underline {X})$ .

We regard $\mathcal {L}(W, f_{1/2}(\underline {X}), \overline {\phi _0})$ as a formal power series of $X_1^{\pm 1}, \dots , X_{n-1}^{\pm 1}$ , or an element of $\mathbb {C}[T_1,\dots ,T_{n-2}] [[T_{n-1}^{\pm 1}]]$ . For $\lambda = (\lambda _1,\dots , \lambda _{n-1}) \in \mathbb {Z}^{n-1}$ , we set $|\lambda | = \lambda _1+\dots +\lambda _{n-1}$ . The following is a key lemma.

Lemma 4.4. Let $W \in \mathcal {W}(\pi , \psi _E)^{K_{2m}^W}$ . Write

$$ \begin{align*} \mathcal{L}(W, f_{1/2}(\underline{X}), \overline{\phi_0}) &= \sum_{\lambda \in \mathbb{Z}^{n-1}} a_\lambda(W) X_1^{\lambda_1}\cdots X_{n-1}^{\lambda_{n-1}} = \sum_{d \in \mathbb{Z}} \mathcal{L}_d(W) T_{n-1}^d \end{align*} $$

with $a_\lambda (W) \in \mathbb {C}$ and $\mathcal {L}_d(W) \in \mathbb {C}[T_1, \dots , T_{n-2}]$ . Then

• • $a_\lambda (W) = 0$ unless $|\lambda | \geq -(n-1)m$ ; and

• • $\mathcal {L}_d(W) = 0$ unless $d \geq -m$ .

Proof. For row vectors $x, u \in E^{n-1}$ and $a \in \mathrm {GL}_{n-1}(E)$ , we put $k(x,a,u)$ to be the matrix

$$\begin{align*}(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a))^{-1} \left( \begin{array}{cc|cc} \mathbf{1}_{n-1} & {}^tu & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \hline 0 & 0 & 1 & -\overline{u}w_{n-1} \\ 0 & 0 & 0 & \mathbf{1}_{n-1} \end{array} \right) w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a). \end{align*}$$

By an easy calculation, $k(x,a,u)$ is equal to

$$\begin{align*}\left( \begin{array}{cc|cc} 1-x{}^tu & -x{}^tuxa & 0 & 0 \\ a^{-1}{}^tu & \mathbf{1}_{n-1} + a^{-1}{}^tuxa & 0 & 0 \\ \hline 0 & 0 & \mathbf{1}_{n-1} -w_{n-1}{}^t\overline{a}{}^t\overline{x} \overline{u} {}^t\overline{a}^{-1}w_{n-1} & w_{n-1}{}^t\overline{a}{}^t\overline{x} \overline{u}{}^t\overline{x}\\ 0 & 0 & -\overline{u}{}^t\overline{a}^{-1}w_{n-1} & 1+\overline{u}{}^t\overline{x} \end{array} \right). \end{align*}$$

In particular, if $xa \in \mathfrak {o}_E^{n-1}$ and $u{}^ta^{-1} \in (\mathfrak {p}_E^{m})^{n-1}$ , then $x{}^tu \in \mathfrak {p}_E^{m}$ so that $k(x,a,u) \in K_{2m}^W$ .

As functions on $g \in G_{n-1}$ , all of $W(w_{1,n-1}\mathbf {v}(x,0;0)g)$ , $f_{s}(g,\mathbf {1}_{n-1}; \underline {X})$ and $\overline {\omega _{\psi }(g)\phi _0}$ are right $K^{W_0}$ -invariant. Hence, by the integral formula with respect to the Iwasawa decomposition, we can write $\mathcal {L}(W,f_{s}(\underline {X}), \overline {\phi _0})$ as

$$\begin{align*}\int_{T_{n-1}}\int_{E^{n-1}} W(w_{1,n-1}\mathbf{v}(x,0;0)t) f_{s}(t, \mathbf{1}_{n-1}; \underline{X}) \overline{\omega_{\psi}(t)\phi_0(x)} \delta_{B_{n-1}}^{-1}(t)dxdt, \end{align*}$$

where $B_{n-1} = T_{n-1}N_{n-1}$ is the upper triangular Borel subgroup of $G_{n-1}$ with the diagonal torus $T_{n-1}$ . Write $t = \mathbf {m}(a)$ with $a = \mathrm {diag}(a_1,\dots ,a_{n-1})$ being a diagonal matrix in $\mathrm {GL}_{n-1}(E)$ . Then $\omega _{\psi }(\mathbf {m}(a))\phi _0(x) \not = 0 \iff xa \in \mathfrak {o}_E^{n-1}$ . In this case, if $W(w_{1,n-1}\mathbf {v}(x,0;0)\mathbf {m}(a)) \not = 0$ , then for $u = (u_1,\dots ,u_{n-1}) \in E^{n-1}$ such that $u{}^ta^{-1} \in (\mathfrak {p}_E^{m})^{n-1}$ , we have

$$ \begin{align*} 0 &\not= W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a)) \\&= W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a) \cdot k(x,a,u)) \\&= W\left( \left( \begin{array}{cc|cc} \mathbf{1}_{n-1} & {}^tu & 0 & 0 \\ 0 & 1 & 0 & 0 \\ \hline 0 & 0 & 1 & -\overline{u}w_{n-1} \\ 0 & 0 & 0 & \mathbf{1}_{n-1} \end{array} \right) w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a) \right) \\&= \psi_E(u_{n-1})W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a)). \end{align*} $$

This shows that

$$\begin{align*}u_{n-1} \in \mathfrak{p}_E^{\mathrm{ord}(a_{n-1})+m} \implies \psi_E(u_{n-1}) = 1. \end{align*}$$

This means that $\mathrm {ord}(a_{n-1})+m \geq 0$ .

Recall that $f_{s}(\mathbf {m}(a), \mathbf {1}_{n-1}; \underline {X}) = \delta _{Q_{n-1}}^{\frac {1}{2}}(\mathbf {m}(a)) W(a; q^{1-2s}\underline {X})$ . By a similar (and well-known) argument, if $W(a; q^{1-2s}\underline {X}) \not = 0$ , then $\mathrm {ord}(a_1) \geq \dots \geq \mathrm {ord}(a_{n-1})$ . Hence, we conclude that if

$$\begin{align*}W(w_{1,n-1}\mathbf{v}(x,0;0)\mathbf{m}(a)) W(a; \underline{X}) \overline{\omega_{\psi}(\mathbf{m}(a))\phi_0(x)} \not= 0, \end{align*}$$

then

$$\begin{align*}\mathrm{ord}(a_1) \geq \dots \geq \mathrm{ord}(a_{n-1}) \geq -m \end{align*}$$

so that

$$\begin{align*}\mathrm{ord}(\det(a)) = \sum_{i=1}^{n-1}\mathrm{ord}(a_i) \geq -(n-1)m. \end{align*}$$

Since the Casselman–Shalika formula [Reference Casselman and Shalika6] tells us that

$$\begin{align*}W(a; \underline{X}) \in \left( \bigoplus_{\substack{\lambda \in \mathbb{Z}^{n-1} \\ |\lambda| = \mathrm{ord}(\det(a))}} \mathbb{C} X_1^{\lambda_1}\cdots X_{n-1}^{\lambda_{n-1}} \right) \cap \mathbb{C}[T_1,\dots,T_{n-2}]T_{n-1}^{\mathrm{ord}(a_{n-1})}, \end{align*}$$

we obtain the assertions.

For $W \in \mathcal {W}(\pi , \psi _E)^{K_{2m}^W}$ , we define $\Psi (W; \underline {X})$ by

$$\begin{align*}\Psi(W; \underline{X}) = \frac{\prod_{i=1}^{n-1}P_\pi(-q^{-1}X_i)\mathcal{L}(W, f_{1/2}(\underline{X}), \phi_0)} {\prod_{i=1}^{n-1}(1-q^{-1}X_i)\prod_{1\leq i < j \leq n-1}(1-q^{-2}X_iX_j)}. \end{align*}$$

Proposition 4.5. If $2m < c(\phi _\pi )$ , then $\Psi (W; \underline {X}) = 0$ for $W \in \mathcal {W}(\pi , \psi _E)^{K_{2m}^W}$ . If $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ , then

$$\begin{align*}\dim_{\mathbb{C}} \left\{ \Psi(W; \underline{X}) \;\middle|\; W \in \mathcal{W}(\pi, \psi_E)^{K_{2m}^W}\right\} \leq 1. \end{align*}$$

Proof. Since $P_\pi (X)$ is a polynomial of X with $P_\pi (0) = 1$ , and since $(1-q^{-1}X_i)^{-1} = \sum _{k=0}^\infty (q^{-1}X_i)^k$ and $(1-q^{-2}X_iX_j)^{-1} = \sum _{k=0}^\infty (q^{-2}X_iX_j)^k$ , if we write

$$ \begin{align*} \Psi(W; \underline{X}) &= \sum_{\lambda \in \mathbb{Z}^{n-1}} \alpha_\lambda(W) X_1^{\lambda_1}\cdots X_{n-1}^{\lambda_{n-1}} = \sum_{d \in \mathbb{Z}} \Psi_d(W; \underline{X}) T_{n-1}^d \end{align*} $$

with $\alpha _\lambda (W) \in \mathbb {C}$ and $\Psi _d(W; \underline {X}) \in \mathbb {C}[T_1, \dots , T_{n-2}]$ , by Lemma 4.4, we see that

• • $\alpha _\lambda (W) = 0$ unless $|\lambda | \geq -(n-1)m$ ; and

• • $\Psi _d(W; \underline {X}) = 0$ unless $d \geq -m$ .

Write $\underline {X}^{-1} = (X_1^{-1}, \dots , X_{n-1}^{-1})$ . By the functional equation (Theorem 4.2 (3), (5)) together with Lemma 4.3, we see that

(*) $$ \begin{align} T_{n-1}^{-c(\phi_\pi)} \Psi(W; \underline{X}^{-1}) = \varepsilon_0 \Psi(W; \underline{X}) \end{align} $$

with

$$\begin{align*}\varepsilon_0 = ((-1)^{c(\phi_\pi)} \varepsilon \cdot \omega_\pi(-1))^{n-1}. \end{align*}$$

The left-hand side and the right-hand side of $(\ast )$ belong to

$$\begin{align*}\bigoplus_{d \leq m-c(\phi_\pi)} \mathbb{C}[T_1, \dots, T_{n-2}] T_{n-1}^{d}, \quad \bigoplus_{d \geq -m} \mathbb{C}[T_1, \dots, T_{n-2}] T_{n-1}^{d}, \end{align*}$$

respectively. Hence, if $\Psi _d(W; \underline {X}) \not = 0$ , then $-m \leq d \leq m-c(\phi _\pi )$ so that $2m \geq c(\phi _\pi )$ . A similar argument shows that if $\alpha _{\lambda }(W) \not = 0$ , then

$$\begin{align*}-(n-1)m \leq |\lambda| \leq (n-1)(m-c(\phi_\pi)). \end{align*}$$

Now we assume that $2m = c(\phi _\pi )$ . Then $\Psi _d(W; \underline {X}) = 0$ unless $d = -m$ . Hence,

$$\begin{align*}T_{n-1}^{m} \Psi(W; \underline{X}) \in \mathbb{C}[T_1, \dots, T_{n-2}] \subset \mathbb{C}[X_1, \dots, X_{n-1}]. \end{align*}$$

This implies that $\alpha _{\lambda }(W) = 0$ unless $\lambda _i \geq -m$ for any $1 \leq i \leq n-1$ . However, since $\alpha _{\lambda }(W) = 0$ unless $|\lambda | = -(n-1)m$ , we see that $\alpha _{\lambda }(W) = 0$ unless $\lambda _1 = \dots = \lambda _{n-1} = -m$ . This means that

$$\begin{align*}\Psi(W; \underline{X}) \in \mathbb{C} T_{n-1}^{-m} \end{align*}$$

so that

$$\begin{align*}\dim_{\mathbb{C}} \left\{ \Psi(W; \underline{X}) \;\middle|\; W \in \mathcal{W}(\pi, \psi_E)^{K_{c(\phi_\pi)}^W}\right\} \leq 1. \end{align*}$$

Next, we assume that $2m = c(\phi _\pi )+1$ . Then $\Psi _d(W; \underline {X}) = 0$ unless $d = -m, -m+1$ , and $\alpha _\lambda (W) = 0$ unless $|\lambda | = -(n-1)m, -(n-1)(m-1)$ . In particular, $\Psi _{-m+1}(W; \underline {X})$ is a scalar so that

$$\begin{align*}\Psi_{-m+1}(W; \underline{X}^{-1}) = \Psi_{-m+1}(W; \underline{X}). \end{align*}$$

By the functional equation $(\ast )$ , we have

$$ \begin{align*} \Psi_{-m+1}(W; \underline{X}^{-1}) &= \varepsilon_0 \Psi_{-m}(W; \underline{X}), \\ \Psi_{-m}(W; \underline{X}^{-1}) &= \varepsilon_0 \Psi_{-m+1}(W; \underline{X}). \end{align*} $$

Hence, $\Psi _{-m}(W; \underline {X})$ is also a scalar. Therefore,

$$\begin{align*}\Psi(W; \underline{X}) \in \mathbb{C}(T_{n-1}^{-m} + \varepsilon_0 T_{n-1}^{-m+1}) \end{align*}$$

so that

$$\begin{align*}\dim_{\mathbb{C}} \left\{ \Psi(W; \underline{X}) \;\middle|\; W \in \mathcal{W}(\pi, \psi_E)^{K_{c(\phi_\pi)+1}^W}\right\} \leq 1. \end{align*}$$

This completes the proof.

By Proposition 3.3, we see that $\mathcal {W}(\pi , \psi _E)^{K_{2m}^W} \ni W \mapsto \Psi (W; \underline {X})$ gives an injective linear map

$$\begin{align*}\Psi \colon \pi_{\psi}^{K_{2m}^W} \hookrightarrow \mathcal{T}. \end{align*}$$

Hence, by Proposition 4.5, we have

• • $\pi _{\psi }^{K_{2m}^W} = 0$ if $2m < c(\phi _\pi )$ ; and

• • $\dim _{\mathbb {C}}(\pi _{\psi }^{K_{2m}^W}) \leq 1$ if $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ .

This completes the proof of Theorem 2.2 (2).

5.1 Theta correspondence

Recall that $V = V_{2n+1}$ (resp. $W = W_{2n}$ ) is a hermitian (resp. skew-hermitian) space over E of dimension $2n+1$ (resp. $2n$ ). Then $\mathbb {W} = V \otimes _E W$ forms a symplectic space of dimension $4n(2n+1)$ equipped with the symplectic form

$$\begin{align*}\left\langle v \otimes w, v' \otimes w' \right\rangle = \mathrm{tr}_{E/F}\left( \left\langle v,v' \right\rangle_V \cdot \left\langle w,w' \right\rangle_W \right). \end{align*}$$

Here, $\mathrm {U}(V)$ , $\mathrm {U}(W)$ and $\mathrm {Sp}(\mathbb {W})$ act on V, W and $\mathbb {W}$ , respectively, all from the left. We have a canonical map $\mathrm {U}(V) \times \mathrm {U}(W) \rightarrow \mathrm {Sp}(\mathbb {W})$ .

Recall that $\chi $ is the unique nontrivial quadratic unramified character of $E^\times $ . Note that $\chi |_{F^\times }$ is equal to the quadratic character corresponding to $E/F$ . Let $\widetilde {\mathrm {Sp}}(\mathbb {W})$ be the metaplectic $\mathbb {C}^\times $ -cover. Using the pair $(\chi _V, \chi _W) = (\chi ^{2n+1}, \chi ^{2n})$ , we have Kudla’s splitting [Reference Kudla15]

$$\begin{align*}\mathrm{U}(V) \times \mathrm{U}(W) \rightarrow \widetilde{\mathrm{Sp}}(\mathbb{W}). \end{align*}$$

Let $\omega _\psi $ be the Weil representation of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ associated to the additive character $\psi $ . By the pullback, we obtain the Weil representation $\omega _{\psi , V, W}$ of $\mathrm {U}(V) \times \mathrm {U}(W)$ . For an irreducible representation $\pi $ of $\mathrm {U}(W)$ , it is known that the maximal $\pi $ -isotypic quotient of $\omega _{\psi , V, W}$ is of the form

$$\begin{align*}\Theta_\psi(\pi) \boxtimes \pi \end{align*}$$

for a smooth representation $\Theta _\psi (\pi )$ of $\mathrm {U}(V)$ of finite length. The Howe duality conjecture, proven by Waldspurger [Reference Waldspurger24], asserts that if $\Theta _\psi (\pi )$ is nonzero, then it has a unique irreducible quotient $\theta _\psi (\pi )$ . We call $\theta _\psi (\pi )$ the theta lift of $\pi $ .

The following is a special case of Prasad’s conjecture, which was proven by Gan–Ichino [Reference Gan and Ichino10]. See also Theorem 4.4 in that paper.

Theorem 5.1. Let $\pi $ be an irreducible $\psi _E$ -generic representation of $\mathrm {U}(W)$ with L-parameter $\phi _\pi $ . Then $\Theta _\psi (\pi )$ is always nonzero. Moreover, $\sigma = \theta _\psi (\pi )$ is generic, and its L-parameter is given by

$$\begin{align*}\phi_\sigma = \phi_\pi\chi \oplus \mathbf{1}, \end{align*}$$

where $\phi _\pi \chi = \phi _\pi \otimes \chi $ .

In particular, if $\sigma = \theta _\psi (\pi )$ , then we have $c(\phi _\sigma ) = c(\phi _\pi )$ and $\omega _{\sigma } = \omega _{\pi }$ . Moreover, if $\pi $ is tempered, then so is $\sigma $ , so that we have $\sigma ^{K_{2m}^V} \not = 0$ for $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ by Theorem 2.1.

5.2 Lattice model

First, we will show that $\pi ^{K_{2m}^W} \not = 0$ . To do this, we use a lattice model $\mathcal {S} = \mathcal {S}(A)$ of the Weil representation $\omega _\psi $ of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ . In this subsection, we recall this model.

Let $\mathbb {W}$ be a symplectic space over F of dimension $2N$ equipped with a symplectic form $\left \langle \cdot , \cdot \right \rangle $ . The group law of the Heisenberg group $H(\mathbb {W}) = \mathbb {W} \oplus F$ is given by

$$\begin{align*}(w_1,t_1) \cdot (w_2, t_2) = \left(w_1+w_2, t_1+t_2+\frac{1}{2}\left\langle w_1,w_2 \right\rangle\right), \end{align*}$$

whose center is $\{0\} \oplus F \cong F$ . By the Stone–von Neumann theorem, there is a unique (up to isomorphism) irreducible admissible representation $(\rho _\psi , \mathcal {S})$ of $H(\mathbb {W})$ whose central character is $\psi $ . The symplectic group $\mathrm {Sp}(\mathbb {W})$ acts on $H(\mathbb {W})$ by $g \cdot (w,t) = (gw,t)$ . By the uniqueness, for $g \in \mathrm {Sp}(\mathbb {W})$ , we have $M_g \in \mathrm {Aut}(\mathcal {S})$ such that

(⋆) $$ \begin{align} M_g \circ \rho_\psi(h) \circ M_g^{-1} = \rho_\psi(gh) \quad\text{for } h \in H(\mathbb{W}). \end{align} $$

By Schur’s lemma, such $M_g$ is determined uniquely up to a nonzero scalar. Define the metaplectic $\mathbb {C}^\times $ -cover $\widetilde {\mathrm {Sp}}(\mathbb {W})$ of $\mathrm {Sp}(\mathbb {W})$ by

$$\begin{align*}\widetilde{\mathrm{Sp}}(\mathbb{W}) = \{(g,M_g) \in \mathrm{Sp}(\mathbb{W}) \times \mathrm{Aut}(\mathcal{S}) \;|\; M_g \text{ satisfies } (\star)\}. \end{align*}$$

We have an exact sequence

given by $\alpha (z) = (\mathbf {1}_{\mathbb {W}}, z \cdot \mathrm {id}_{\mathcal {S}})$ and $\beta (g,M_g) = g$ . The Weil representation $\omega _\psi $ of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ on the space $\mathcal {S}$ is defined by

$$\begin{align*}\omega_\psi(g,M_g) = M_g. \end{align*}$$

Now we shall give a realization of the space $\mathcal {S}$ . Let A be a lattice of $\mathbb {W}$ (i.e., a free $\mathfrak {o}_F$ -submodule of rank $2N$ ). The dual lattice $A^*$ is defined by

$$\begin{align*}A^* = \left\{w \in \mathbb{W} \;\middle|\; \left\langle w,a \right\rangle \in \mathfrak{o}_F \text{for any} a \in A \right\}. \end{align*}$$

Suppose that A is self-dual (i.e., $A^* = A$ ). Let $\mathcal {S}(A)$ be the space of locally constant, compactly supported functions $\phi \colon H(\mathbb {W}) \rightarrow \mathbb {C}$ such that

$$\begin{align*}\phi((a,t) \cdot h) = \psi(t) \phi(h) \end{align*}$$

for $(a,t) \in A \oplus F$ and $h \in H(\mathbb {W})$ . The group $H(\mathbb {W})$ acts on $\mathcal {S}(A)$ by the right translation $\rho _\psi $ . It is known that the representation $(\rho _\psi , \mathcal {S}(A))$ of $H(\mathbb {W})$ is irreducible with the central character $\psi $ . This gives a realization $(\omega _\psi , \mathcal {S}(A))$ of the Weil representation which is called a lattice model. Since $(a,0) \cdot (w,0) = (a+w, \frac {1}{2}\left \langle a,w \right \rangle )$ , by the restriction to $\mathbb {W} \oplus \{0\}$ , we can identify $\mathcal {S}(A)$ with the space of locally constant, compactly supported functions $\phi \colon \mathbb {W} \rightarrow \mathbb {C}$ such that

$$\begin{align*}\phi(a+w) = \psi\left(-\frac{1}{2}\left\langle a,w \right\rangle\right) \phi(w) \end{align*}$$

for $a \in A$ and $w \in \mathbb {W}$ .

For $g \in \mathrm {Sp}(\mathbb {W})$ , we define $M[g] \in \mathrm {Aut}(\mathcal {S}(A))$ by

$$\begin{align*}(M[g]\phi)(w) = \int_{A} \psi\left(\frac{1}{2}\left\langle a,w \right\rangle\right) \phi(g^{-1} \cdot (a+w)) da \end{align*}$$

for $\phi \in \mathcal {S}(A)$ and $w \in \mathbb {W}$ . Here, $da$ is the Haar measure on A normalized so that $\mathrm {vol}(A) = 1$ . It is easy to check that $(g,M[g]) \in \widetilde {\mathrm {Sp}}(\mathbb {W})$ .

Let $K_A$ be the stabilizer of A in $\mathrm {Sp}(\mathbb {W})$ . Then we have

$$\begin{align*}(M[k]\phi)(w) = \phi(k^{-1} \cdot w) \end{align*}$$

for $k \in K_A$ , $\phi \in \mathcal {S}(A)$ , and $w \in \mathbb {W}$ . The map $k \mapsto (k,M[k])$ gives a splitting $K_A \rightarrow \widetilde {\mathrm {Sp}}(\mathbb {W})$ . If we identify $K_A$ with the image, the restriction of the Weil representation $(\omega _\psi , \mathcal {S}(A))$ to $K_A$ is given by $\omega _\psi (k)\phi (w) = \phi (k^{-1} \cdot w)$ .

5.3 Families of lattices

Take bases $\{e_n, \dots , e_1, e_0, e_{-1}, \dots , e_{-n}\}$ of V and $\{f_n,\dots ,f_1, f_{-1},\dots ,f_{-n}\}$ of W, respectively, as in §2.1. Set

$$ \begin{align*} \Gamma_V &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_i\right) \oplus \mathfrak{o}_E e_0 \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_{-i} \right), \\ \Gamma_W &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E f_i\right) \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E f_{-i} \right). \end{align*} $$

Then $\Gamma _V$ and $\Gamma _W$ are self-dual lattices (i.e., $\Gamma _V^* = \Gamma _V$ and $\Gamma _W^* = \Gamma _W$ ).

In this subsection, for two $\mathfrak {o}_E$ -modules $\Gamma _1$ and $\Gamma _2$ , we denote by $\Gamma _1 \otimes \Gamma _2$ the tensor product of $\mathfrak {o}_E$ -modules. We put

$$\begin{align*}A = \Gamma_V \otimes \Gamma_W. \end{align*}$$

This is a self-dual lattice of $\mathbb {W} = V \otimes _F W$ , (i.e., $A^* = A$ ). We will consider the lattice model $(\omega _\psi , \mathcal {S}(A))$ of the Weil representation of $\widetilde {\mathrm {Sp}}(\mathbb {W})$ .

Fix a non-negative even integer $2m \geq 0$ . We consider lattices

$$ \begin{align*} M_{2m} &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_i\right) \oplus \mathfrak{p}_E^{m} e_0 \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_{-i}\right),\\ N_{2m} &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E f_i \right) \oplus \left(\bigoplus_{i=1}^{n-1} \mathfrak{o}_E f_{-i} \right) \oplus \mathfrak{p}_E^{m} f_{-n} \end{align*} $$

of V and W, respectively. Then $M_{2m} \subset \Gamma _V$ and $N_{2m} \subset \Gamma _W$ . Moreover, the dual lattices are given by

$$ \begin{align*} M_{2m}^* &= \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_i\right) \oplus \mathfrak{p}_E^{-m} e_0 \oplus \left(\bigoplus_{i=1}^n \mathfrak{o}_E e_{-i}\right), \\ N_{2m}^* &= \mathfrak{p}_E^{-m} f_n \oplus \left(\bigoplus_{i=1}^{n-1} \mathfrak{o}_E f_i \right) \oplus \left(\bigoplus_{i=1}^{n} \mathfrak{o}_E f_{-i} \right). \end{align*} $$

Recall that in Section 2.4, we defined compact subgroups $K_{2m}^V$ and $K_{2m}^W$ of $\mathrm {U}(V)$ and $\mathrm {U}(W)$ , respectively. The following lemma is easy to check.

Lemma 5.2. We have

$$ \begin{align*} K_{2m}^V = \{h \in \mathrm{U}(V) \;|\; (h-1) \cdot M_{2m}^* \subset M_{2m} \}, \\ K_{2m}^W = \{g \in \mathrm{U}(W) \;|\; (g-1) \cdot N_{2m}^* \subset N_{2m} \}. \end{align*} $$

In particular, $K_{2m}^V \times K_{2m}^W$ is contained in $K_A$ under the canonical map $\mathrm {U}(V) \times \mathrm {U}(W) \rightarrow \mathrm {Sp}(\mathbb {W})$ .

Let $\mathcal {S}(A)_{M_{2m}}$ be the subspace of $\mathcal {S}(A)$ consisting of functions $\phi \colon \mathbb {W} \rightarrow \mathbb {C}$ such that $\mathrm {Supp}(\phi ) \subset M_{2m}^* \otimes \Gamma _W$ . We will use the following result proven by Waldspurger.

We will apply this proposition to the compact subgroup $J_{2m}^V$ generated by $K_{2m}^V$ and $E^1 \cap (1+\mathfrak {p}_E^{m})$ , where the latter is regarded as a subgroup of the center of $\mathrm {U}(V)$ . Namely, $J_{0}^V = K_{0}^V$ , and

for $2m> 0$ . It is clear that $J_{2m}^V \supset K_{2m}^V$ .

We check the second and third conditions in Proposition 5.3.

Proof. For $t \in \mathfrak {p}_E^{-m}$ and $w \in \Gamma _W$ , define $\phi _{t,w} \in \mathcal {S}(A)$ so that $\mathrm {Supp}(\phi _{t,w}) = A + te_0 \otimes w$ and $\phi _{t,w}(te_0 \otimes w) = 1$ . Then $\mathcal {S}(A)_{M_{2m}}$ is equal to the $\mathbb {C}$ -span of

$$\begin{align*}\left\{\phi_{t,w} \;\middle|\; t \in \mathfrak{p}_E^{-m},\; w \in \Gamma_W \right\}. \end{align*}$$

For $k \in J_{2m}^{V}$ , write $ke_0 = \sum _{i = -n}^n k_i e_i$ . Then

$$\begin{align*}k_i \in \left\{ \begin{aligned} &\mathfrak{p}_E^{m} &\quad&\text{if } i \not=0, \\ &1+\mathfrak{p}_E^{m} &\quad&\text{if } i = 0. \end{aligned} \right. \end{align*}$$

In particular, we see that $(k-1)te_0 \otimes w \in A$ . Hence,

$$ \begin{align*} \mathrm{Supp}(\omega_\psi(k)\phi_{t,w}) &= k(A + te_0 \otimes w) \\&= A + (k-1)te_0 \otimes w + te_0 \otimes w \\&= A + te_0 \otimes w = \mathrm{Supp}(\phi_{t,w}). \end{align*} $$

Moreover,

$$ \begin{align*} \omega_\psi(k)\phi_{t,w}(te_0 \otimes w) &= \omega_\psi(k)\phi_{t,w}\left(kte_0 \otimes w - (k-1)te_0 \otimes w\right) \\&= \psi\left( \frac{1}{2}\left\langle (k-1)te_0 \otimes w, kte_0 \otimes w \right\rangle \right) \omega_\psi(k)\phi_{t,w}(kte_0 \otimes w) \\&= \psi_E\left( \left\langle (k-1)te_0,kte_0 \right\rangle_V \cdot \left\langle w,w \right\rangle_W\right) \phi_{t,w}(te_0 \otimes w) \\&= \psi_E\left( N_{E/F}(t)(\left\langle ke_0,ke_0 \right\rangle_V-\left\langle e_0,ke_0 \right\rangle_V) \cdot \left\langle w,w \right\rangle_W\right) \\&= \psi_E\left( N_{E/F}(t)(1-\overline{k_0}) \left\langle w,w \right\rangle_W \right). \end{align*} $$

Hence, for $t \in \mathfrak {p}_E^{-m}$ , $w \in \Gamma _W$ and $k \in J_{2m}^V$ , there exists $c \in \mathbb {C}^\times $ such that $\omega _\psi (k)\phi _{t,w} = c\phi _{t,w}$ . This shows that $\mathcal {S}(A)_{M_{2m}}$ is stable by $J_{2m}^V$ . Moreover, if $k_0 \in \mathfrak {p}_E^{2m}$ or $\left \langle w,w \right \rangle _W = 0$ , then $c=1$ . Hence, we have $(\mathcal {S}(A)_{M_{2m}})^{K_{2m}^V} = \mathcal {S}(A)_{M_{2m}}$ and $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V} \not = \{0\}$ .

Therefore, by Proposition 5.3, we see that $\mathcal {S}(A)^{J_{2m}^V}$ is generated by $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ as a representation of $\mathrm {U}(W)$ . If $2m> 0$ , then $\mathcal {S}(A)_{M_{2m}} \supset \mathcal {S}(A)_{M_{2m-2}}$ . Let $\mathcal {S}(A)_{M_{2m} \setminus M_{2m-2}}$ be the subspace spanned by

$$\begin{align*}\left\{\phi_{t,w} \;\middle|\; \mathrm{ord}(t) = -m,\; w \in \Gamma_W \setminus \varpi\Gamma_W \right\}. \end{align*}$$

Then we have

$$\begin{align*}\mathcal{S}(A)_{M_{2m}} = \mathcal{S}(A)_{M_{2m-2}} \oplus \mathcal{S}(A)_{M_{2m} \setminus M_{2m-2}}. \end{align*}$$

Lemma 5.5. Suppose that $2m> 0$ . The image $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ under the projection $\mathcal {S}(A)_{M_{2m}} \twoheadrightarrow \mathcal {S}(A)_{M_{2m} \setminus M_{2m-2}}$ is equal to the one of the subspace spanned by

$$\begin{align*}\left\{ \omega_\psi(k')\phi_{t, f_n} \;\middle|\; \mathrm{ord}(t) = -m,\; k' \in K_0^W \right\}. \end{align*}$$

Moreover, $\phi _{t,f_n}$ is fixed by $K_{2m}^W$ , and $\phi _{t,f_{-n}}$ is fixed by ${}^tK_{2m}^W$ .

Proof. As we have seen in the proof of Lemma 5.4, $k \in J_{2m}^V$ acts on $\phi _{t,w}$ by the character

Hence, the image in question is equal to the one of the subspace spanned by $\phi _{t,w}$ with $\mathrm {ord}(t) = -m$ and $w \in \Gamma _W \setminus \varpi \Gamma _W$ such that $\left \langle w,w \right \rangle _W \in \mathfrak {p}_E^m$ . It means that

$$\begin{align*}\left\langle w,w \right\rangle_W \equiv \left\langle f_n, f_n \right\rangle_W \bmod \mathfrak{p}_E^{m}. \end{align*}$$

Note that

$$\begin{align*}K_0^W = \{g \in \mathrm{U}(V) \;|\; g \Gamma_W = \Gamma_W\} \end{align*}$$

is a hyperspecial maximal compact subgroup of $\mathrm {U}(W)$ . Hence, there exists $k' \in K_0^W$ such that $w \equiv k' \cdot f_n \bmod \varpi ^{m}\Gamma _W$ . In particular, we have

$$\begin{align*}te_0 \otimes w - te_0 \otimes k' \cdot f_n \in A. \end{align*}$$

Hence, we can find $c \in \mathbb {C}^\times $ such that $\phi _{t,w} = c\phi _{t, k' \cdot f_n} = c \cdot \omega _\psi (k') \phi _{t,f_n}$ . This shows the first assertion.

Fix $k' \in K_{2m}^W$ . Since $(k'-1)f_n \in \varpi ^{m} \Gamma _W$ , we have $(k'-1)(te_0 \otimes f_n) \in A$ for $t \in \mathfrak {p}_E^{-m}$ . Hence, $\mathrm {Supp}(\omega _\psi (k') \phi _{t,f_n}) = \mathrm {Supp}(\phi _{t,f_n})$ . Moreover,

$$ \begin{align*} \omega_\psi(k') \phi_{t,f_n}(te_0 \otimes f_n) &= \omega_\psi(k') \phi_{t,f_n}(k'(te_0 \otimes f_n) - (k'-1)(te_0 \otimes f_n)) \\&= \psi\left( \frac{1}{2}\left\langle (k'-1)(te_0 \otimes f_n), k'(te_0 \otimes f_n) \right\rangle \right) \omega_\psi(k') \phi_{t,f_n}(k'(te_0 \otimes f_n)) \\&= \psi_E\left( N_{E/F}(t) \left\langle (k'-1)f_n,k'f_n \right\rangle_W \right). \end{align*} $$

Since $\left \langle (k'-1)f_n,k'f_n \right \rangle _W = \left \langle -f_n,k'f_n \right \rangle _W \in \mathfrak {p}_E^{2m}$ , we have $\omega _\psi (k') \phi _{t,f_n}(te_0 \otimes f_n) = 1$ . Therefore, we conclude that $\omega _\psi (k') \phi _{t,f_n} = \phi _{t,f_n}$ for $k' \in K_{2m}^W$ . By a similar calculation, one can prove that $\omega _\psi (k') \phi _{t,f_{-n}} = \phi _{t,f_{-n}}$ for $k' \in {}^tK_{2m}^W$ . This completes the proof.

5.4 Existence of $K_{2m}^W$ -fixed vectors

Let $\pi $ be an irreducible $\psi _E$ -generic tempered representation of $\mathrm {U}(W)$ with the L-parameter $\phi _\pi $ and the central character $\omega _\pi $ . Consider its theta lift $\sigma = \theta _\psi (\pi )$ . It is an irreducible generic tempered representation of $\mathrm {U}(V)$ with L-parameter $\phi _\sigma = \phi _\pi \chi \oplus \mathbf {1}$ . In particular, $c(\phi _\sigma ) = c(\phi _\pi )$ so that $\sigma ^{K_{2m}^V} \not = 0$ for $2m \geq c(\phi _\pi )$ by Theorem 2.1. Since $\omega _\sigma = \omega _\pi $ , we see that $\sigma ^{J_{2m}^V} \not = 0$ if $2m \geq c(\phi _\pi )$ and $\omega _\pi |_{1+\mathfrak {p}_E^m} = \mathbf {1}$ .

Set $\omega _\psi = \omega _{\psi ,V,W}$ . By the definition of theta lifts, we have a $\mathrm {U}(V) \times \mathrm {U}(W)$ -equivariant surjective map

$$\begin{align*}\Phi \colon \omega_\psi \twoheadrightarrow \sigma \boxtimes \pi. \end{align*}$$

Proof. We realize $\omega _\psi $ on the lattice model $\mathcal {S}(A)$ . Since $\Sigma \mapsto \Sigma ^{J_{2m}^V}$ is an exact functor on the category of smooth representations $\Sigma $ of $\mathrm {U}(V)$ , we obtain a $\mathrm {U}(W)$ -equivariant surjective map

$$\begin{align*}\Phi \colon \mathcal{S}(A)^{J_{2m}^V} \twoheadrightarrow \sigma^{J_{2m}^V} \boxtimes \pi. \end{align*}$$

By Proposition 5.3 together with Lemma 5.4, its restriction to $(\mathcal {S}(A)_{M_{2m}})^{J_{2m}^V}$ is still nonzero. Since $\sigma ^{K_{2m-2}^V} = 0$ , this map factors through the restriction of the projection $\mathcal {S}(A)_{M_{2m}} \twoheadrightarrow \mathcal {S}(A)_{M_{2m} \setminus M_{2m-2}}$ . Hence, by Lemma 5.5, there exists $t \in E^\times $ with $\mathrm {ord}(t) = -m$ such that $\Phi (\phi _{t,f_n}) \not = 0$ . Since $\phi _{t,f_n}$ is fixed by $J_{2m}^V \times K_{2m}^W$ , we have $\Phi (\phi _{t,f_n}) \in \sigma ^{J_{2m}^V} \boxtimes \pi ^{K_{2m}^W}$ so that $\pi ^{K_{2m}^W} \not = 0$ . By the same argument, one can show that $\Phi (\phi _{t,f_{-n}}) \not = 0$ for some $t \in \mathfrak {p}_E^{-m}$ .

5.5 Proof of Theorem 2.2 (3)

The goal of the rest of this section is to show that $\pi _\psi ^{K_{2m}^W} \not = 0$ if $2m = c(\phi _\pi )$ or $2m = c(\phi _\pi )+1$ and if $\omega _\pi $ is trivial on $1+\mathfrak {p}_E^m$ . If $2m = c(\phi _\pi ) = 0$ , then $\pi $ is unramified (with respect to the hyperspecial maximal compact subgroup $K_0^W$ ), and the Casselman–Shalika formula [Reference Casselman and Shalika6] shows that $\pi _\psi ^{K_{0}^W} \not = 0$ . See Remark 4.1. Hence, we may assume that $c(\phi _\pi )> 0$ so that $2m> 0$ .

We need further notations. Set

$$\begin{align*}X = \bigoplus_{i=1}^n Ee_i,\quad V_0 = E e_0, \quad X^* = \bigoplus_{i=1}^n Ee_{-i}. \end{align*}$$

Hence, $V = X \oplus V_0 \oplus X^*$ . For $a \in \mathrm {GL}(X)$ , $b \in \mathrm {Hom}(V_0,X)$ and $c \in \mathrm {Hom}(X^*,X)$ , we define $a^* \in \mathrm {GL}(X^*)$ , $b^* \in \mathrm {Hom}(X^*,V_0)$ and $c^* \in \mathrm {Hom}(X^*,X)$ so that

$$\begin{align*}\left\langle ax, x' \right\rangle_V = \left\langle x,a^*x' \right\rangle_V, \quad \left\langle be_0, x' \right\rangle_V = \left\langle e_0,b^*x' \right\rangle_V, \quad \left\langle cx', x" \right\rangle_V = \left\langle x', c^*x" \right\rangle_V \end{align*}$$

for $x \in X$ and $x',x" \in X^*$ . For $a \in \mathrm {GL}(X)$ , $b \in \mathrm {Hom}(V_0,X)$ and

$$\begin{align*}c \in \mathrm{Herm}(X^*,X) = \{c \in \mathrm{Hom}(X^*,X) \;|\; c^*=-c\}, \end{align*}$$

we put

$$ \begin{align*} \mathbf{m}_X(a) &= \begin{pmatrix} a && \\ &\mathbf{1}_{V_0}& \\ && (a^*)^{-1} \end{pmatrix}, \\ \quad \mathbf{n}_1(b) &= \begin{pmatrix} \mathbf{1}_X & b & -\frac{1}{2}bb^*\\ &\mathbf{1}_{V_0}& b^* \\ && \mathbf{1}_{X^*} \end{pmatrix}, \\ \mathbf{n}_2(c) &= \begin{pmatrix} \mathbf{1}_X & & c \\ &\mathbf{1}_{V_0}& \\ && \mathbf{1}_{X^*} \end{pmatrix}. \end{align*} $$

These are elements in $\mathrm {U}(V)$ .

Similarly, set

$$\begin{align*}Y = \bigoplus_{i=1}^n Ef_i, \quad Y^* = \bigoplus_{i=1}^n Ef_{-i} \end{align*}$$

so that $W = Y \oplus Y^*$ . For $a \in \mathrm {GL}(Y)$ and $c \in \mathrm {Hom}(Y^*,Y)$ , we define $a^* \in \mathrm {GL}(X^*)$ and $c^* \in \mathrm {Hom}(Y^*,Y)$ so that

$$\begin{align*}\left\langle ay, y' \right\rangle_W = \left\langle y,a^*y' \right\rangle_W, \quad \left\langle cy', y" \right\rangle_W = \left\langle y', c^*y" \right\rangle_W \end{align*}$$

for $y \in Y$ and $y',y" \in Y^*$ . For $a \in \mathrm {GL}(Y)$ and

$$\begin{align*}c \in \mathrm{Herm}(Y^*,Y) = \{c \in \mathrm{Hom}(Y^*,Y) \;|\; c^*=-c\}, \end{align*}$$

we put

$$\begin{align*}\mathbf{m}_Y(a) = \begin{pmatrix} a & \\ & (a^*)^{-1} \end{pmatrix}, \quad \mathbf{n}(c) = \begin{pmatrix} \mathbf{1}_Y & c \\ & \mathbf{1}_{Y^*} \end{pmatrix}. \end{align*}$$

These are elements in $\mathrm {U}(W)$ .

Define $a_\delta \in \mathrm {GL}(X)$ by

$$\begin{align*}a_\delta \colon e_i \mapsto \delta^{-i} e_i \end{align*}$$

for $-n \leq i \leq n$ . If we fix a nonzero Whittaker functional $l_\sigma \in \mathrm {Hom}_{N_{2n+1}}(\sigma , \psi _E)$ for $\sigma $ , then $l_{\sigma }' = l_\sigma \circ \sigma (\mathbf {m}_X(a_\delta ))$ is a nonzero Whittaker functional with respect to the character $\psi _{E}^\delta \colon N_{2n+1} \rightarrow \mathbb {C}^\times $ given by

$$\begin{align*}\psi_{E}^\delta(u) = \psi_E\left( \delta^{-1} \sum_{i=1}^{n} \left\langle u e_{i-1}, e_{-i} \right\rangle_V \right). \end{align*}$$

This is the generic character considered in [Reference Cheng8].

Now we fix $t \in E^\times $ with $\mathrm {ord}(t) = -m$ such that $\Phi (\phi _{t, f_{-n}}) \not = 0$ . This belongs to $\sigma ^{J_{2m}^V} \boxtimes \pi $ . Note that $\mathbf {m}_X(t \cdot \mathbf {1}_X) K_{2m}^V \mathbf {m}_X(t \cdot \mathbf {1}_X)^{-1}$ is the compact group $K_{n,2m}$ considered in [Reference Cheng8]. In particular, the Whittaker functional

$$\begin{align*}l^{\prime}_{\sigma,t} = l^{\prime}_\sigma \circ \sigma(\mathbf{m}_X(t \cdot \mathbf{1}_X)) = l_\sigma \circ \sigma(\mathbf{m}_X(a_\delta t)) \end{align*}$$

with respect to

$$\begin{align*}\psi_{E,t}^\delta \colon N_{2n+1} \ni u \mapsto \psi_E^{\delta}(\mathbf{m}_X(t \cdot \mathbf{1}_X) \cdot u \cdot \mathbf{m}_X(t \cdot \mathbf{1}_X)^{-1}) \in \mathbb{C}^\times \end{align*}$$

is nonzero on $\sigma ^{K_{2m}^V}$ by [Reference Cheng8, Theorem 1.4, Lemma 7.5]. Therefore, the image $\phi _{t, f_{-n}}$ under the composition of $N_{2n+1} \times \mathrm {U}(W)$ -equivariant maps

is nonzero.

By the same argument as the proof of [Reference Mao and Rallis18, Proposition 2.3], one can prove that the maximal quotient of $\omega _\psi $ on which $N_{2n+1}$ acts by $\psi _{E,t}^\delta $ is isomorphic to the compact induction $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ , where $N_{2n}'$ is the unipotent radical of the Borel subgroup of $\mathrm {U}(W)$ stabilizing the flag

$$\begin{align*}E f_1 \subset E f_1 \oplus E f_2 \subset \dots \subset E f_1 \oplus \dots \oplus E f_n = Y, \end{align*}$$

and $\mu $ is a character of $N_{2n}'$ given by

$$\begin{align*}\mu(u) = \psi_E \left( \sum_{i=1}^n \left\langle u f_{i+1}, f_{-i} \right\rangle + N_{E/F}(t)\left\langle u f_{-n}, f_{-n} \right\rangle \right). \end{align*}$$

Here, we note that $N_{2n}'$ differs from $N_{2n}$ defined in Section 2.2.

Hence, the map

factors through $\omega _\psi \rightarrow \mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ . Namely, we have a nonzero $\mathrm {U}(W)$ -equivariant map

$$\begin{align*}\mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \rightarrow \pi. \end{align*}$$

The following is a key lemma, which will be proven in Section 5.7 below.

Lemma 5.7. Let $\widetilde {F}_{t,f_{-n}} \in \mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ be the image of $\phi _{t, f_{-n}} \in \mathcal {S}(A)$ . Then $\widetilde {F}_{t,f_{-n}}$ is right ${}^tK_{2m}^W$ -invariant and

$$\begin{align*}\mathrm{Supp}(\widetilde{F}_{t,f_{-n}}) = N_{2n}' \cdot {}^tK_{2m}^W. \end{align*}$$

Note that having a $\mathrm {U}(W)$ -equivariant map

$$\begin{align*}\mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \rightarrow \pi \end{align*}$$

is equivalent to giving a $\mathrm {U}(W)$ -equivariant map

$$\begin{align*}\pi^\vee \rightarrow \mathrm{Ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu^{-1}). \end{align*}$$

These are related as follows. Suppose that $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu ) \ni \widetilde {F} \mapsto v \in \pi $ corresponds to $\pi ^\vee \ni v' \mapsto W \in \mathrm {Ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu ^{-1})$ . Then

$$\begin{align*}(v,v')_\pi = \int_{N_{2n}' \backslash \mathrm{U}(W)} \widetilde{F}(g) W(g) dg. \end{align*}$$

By Lemma 5.7, there exists $\widetilde {F} \in \mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ such that

• • its image v in $\pi $ is nonzero;

• • $\widetilde {F}$ is right ${}^tK_{2m}^W$ -invariant;

• • $\mathrm {Supp}(\widetilde {F}) = N_{2n}' \cdot {}^tK_{2m}^W$ .

Hence, $v \in \pi ^{{}^tK_{2m}^W}$ . One can take $v' \in (\pi ^\vee )^{{}^tK_{2m}^W}$ such that $(v,v')_\pi \not = 0$ . Let $W \in \mathrm {Ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu ^{-1})$ be the image of $v'$ . Then W is right ${}^tK_{2m}^W$ -invariant, and

$$\begin{align*}0 \not= (v,v')_\pi = \int_{N_{2n}' \backslash \mathrm{U}(W)} \widetilde{F}(g) W(g) dg = c\widetilde{F}(\mathbf{1}) W(\mathbf{1}) \end{align*}$$

for some constant $c> 0$ . Hence, $W(\mathbf {1}) \not = 0$ . Moreover, since $\mathbf {v}(0,0;z) \in N_{2n}'$ , we have

$$\begin{align*}W(\mathbf{v}(0,0;z)) = \mu^{-1}(\mathbf{v}(0,0;z)) W(\mathbf{1}) = \psi^{-1}(N_{E/F}(t)z) W(\mathbf{1}) \end{align*}$$

for $z \in F$ . Therefore, via $v' \mapsto W \mapsto W(\mathbf {1})$ , we conclude that

$$\begin{align*}(\pi^\vee)^{{}^tK_{2m}^W}_{\psi^{\prime-1}} \not= 0, \end{align*}$$

where we put $\psi '(z) = \psi (N_{E/F}(t)z)$ . Since

$$\begin{align*}{}^tK_{2m}^W = \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ &&\overline{t}^{-1} \end{pmatrix}^{-1} K_{2m}^W \begin{pmatrix} t && \\ &\mathbf{1}_{2n-2}& \\ && \overline{t}^{-1} \end{pmatrix}, \end{align*}$$

as in Section 2.3, we have

$$\begin{align*}(\pi^\vee)^{K_{2m}^W}_{\psi^{-1}} \cong (\pi^\vee)^{{}^tK_{2m}^W}_{\psi^{\prime-1}} \not= 0. \end{align*}$$

Since $\pi $ is $\psi _E$ -generic if and only if $\pi ^\vee $ is $\psi _E^{-1}$ -generic, by replacing $\pi $ and $\psi $ with $\pi ^\vee $ and $\psi ^{-1}$ , respectively, we conclude that

$$\begin{align*}\pi^{K_{2m}^W}_\psi \not= 0. \end{align*}$$

Therefore, Theorem 2.2 (3) is reduced to proving Lemma 5.7.

5.6 Mixed model

To show Lemma 5.7, we review the argument in the proof of [Reference Mao and Rallis18, Proposition 2.3]. For this, we use another model of the Weil representation $\omega _\psi = \omega _{\psi ,V,W}$ of $\mathrm {U}(V) \times \mathrm {U}(W)$ . It is known that the Weil representation $\omega _\psi $ can be realized on the space $\mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ , which is called a mixed model. See, for example, [Reference Gan and Ichino10, Section 7.4]. Let us recall some formulas for the action of $\mathrm {U}(V) \times \mathrm {U}(W)$ on this space.

For $\varphi _1 \otimes \varphi _2 \in \mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ and $(x,y) \in (X^* \otimes W) \times (V_0 \otimes Y^*)$ ,

$$ \begin{align*} &\omega_\psi(g)(\varphi_1 \otimes \varphi_2)(x,y) = \varphi_1(g^{-1}x) \cdot \omega_\psi^0(g)\varphi_2(y), \quad g \in \mathrm{U}(W), \\ &\omega_\psi(h_0)(\varphi_1 \otimes \varphi_2)(x,y) = \varphi_1(x) \cdot \omega_\psi^0(h_0)\varphi_2(y), \quad h_0 \in \mathrm{U}(V_0), \\ &\omega_\psi(\mathbf{m}_X(a))(\varphi_1 \otimes \varphi_2)(x,y) = \chi_W(\det a) |\det a|^n \varphi_1(a^* x) \cdot \varphi_2(y), \quad a \in \mathrm{GL}(X), \\ &\omega_\psi(\mathbf{n}_1(b))(\varphi_1 \otimes \varphi_2)(x,y) = \varphi_1(x) \cdot \rho_\psi^0(b^*x, 0)\varphi_2(y), \quad b \in \mathrm{Hom}(V_0,X), \\ &\omega_\psi(\mathbf{n}_2(c))(\varphi_1 \otimes \varphi_2)(x,y) = \psi\left(\frac{1}{2}\left\langle cx,x \right\rangle\right) \varphi_1(x) \cdot \varphi_2(y), \quad c \in \mathrm{Herm}(X^*,X). \end{align*} $$

Here, $\mathcal {S}(V_0 \otimes Y^*)$ is regarded as the Schrödinger model of

• • the irreducible representation $\rho _\psi ^0$ of the Heisenberg group $H(V_0 \otimes W)$ on $\mathcal {S}(V_0 \otimes Y^*)$ with the central character $\psi $ ; and

• • the Weil representation $\omega _\psi ^0$ of $\mathrm {U}(V_0) \times \mathrm {U}(W)$ .

Hence, for $\varphi _2 \in \mathcal {S}(V_0 \otimes Y^*)$ and $y \in V_0 \otimes Y^*$ , we have

$$\begin{align*}\rho_\psi^0((y_+ + y_-,t))\varphi_2(y) = \psi\left( t + \left\langle y,y_+ \right\rangle + \frac{1}{2}\left\langle y_-,y_+ \right\rangle \right) \varphi_2(y+y_-) \end{align*}$$

for $y_+ \in V_0 \otimes Y$ and $y_- \in V_0\otimes Y^*$ , and

$$ \begin{align*} &\omega_\psi^0(\mathbf{m}_Y(a))\varphi_2(y) = \chi(\det a)|\det a|^{\frac{1}{2}}\varphi(a^*y), \quad a \in \mathrm{GL}(Y), \\ &\omega_\psi^0(\mathbf{n}(c))\varphi_2(y) = \psi\left(\frac{1}{2} \left\langle cy,y \right\rangle\right) \varphi_2(y) \quad c \in \mathrm{Herm}(Y^*,Y). \end{align*} $$

Moreover, $\omega _\psi ^0(J_{2n})\varphi _2$ is given by a Fourier transform of $\varphi _2$ . For more precision, see [Reference Gan and Ichino10, Section 7.4].

For $\varphi _1 \otimes \varphi _2 \in \mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ , define

$$\begin{align*}F_{\varphi_1 \otimes \varphi_2}(g) = \varphi_1(g^{-1} x_0) \cdot \omega_\psi^0(g)\varphi_2(y_0), \end{align*}$$

where we set

$$\begin{align*}x_0 = \sum_{i=1}^n \frac{1}{2\delta} e_{-i} \otimes f_{n+1-i}, \quad y_0 = t e_0 \otimes f_{-n}. \end{align*}$$

Let $Q_{2n} = M_{2n,S}N_{2n,S}$ be the Siegel parabolic subgroup of $\mathrm {U}(W)$ stabilizing Y, where $M_{2n,S} = \{\mathbf {m}_Y(a) \;|\; a \in \mathrm {GL}(Y)\}$ is its Levi subgroup, and $N_{2n,S}$ is its unipotent radical. Note that $N_{2n,S} \subset N_{2n}'$ . We regard $\mu $ as a character of $N_{2n,S}$ by the restriction. For $u \in N_{2n,S}$ , since $u^{-1}x_0 = x_0$ and

$$ \begin{align*} \psi\left(\frac{1}{2}\left\langle uy_0,y_0 \right\rangle\right) &=\psi_E\left(\left\langle te_0, te_0 \right\rangle_V \left\langle uf_{-n},f_{-n} \right\rangle_W \right) \\&= \psi_E\left(N_{E/F}(t) \left\langle uf_{-n},f_{-n} \right\rangle_W \right) = \mu(u), \end{align*} $$

we see that $F_{\varphi _1 \otimes \varphi _2}(g) \in \mathrm {ind}_{N_{2n,S}}^{\mathrm {U}(W)}(\mu )$ . Note that $\mathbf {n}_2(c)$ acts trivially on $F_{\varphi _1 \otimes \varphi _2}$ for $c \in \mathrm {Herm}(X^*,X)$ since Y is totally isotropic. However, for $b \in \mathrm {Hom}(V_0,X)$ , since $\mathbf {n}_1(b)$ commutes with $g \in \mathrm {U}(W)$ , we see that

$$ \begin{align*} F_{\omega_\psi(\mathbf{n}_1(b))(\varphi_1 \otimes \varphi_2)}(g) &= \omega_\psi(g) \circ \omega_\psi(\mathbf{n}_1(b)) (\varphi_1\otimes \varphi_2) (x_0,y_0) \\&= \omega_\psi(\mathbf{n}_1(b)) \circ \omega_\psi(g) (\varphi_1\otimes \varphi_2) (x_0,y_0) \\&= \rho_\psi^0(b^*x_0,0) \circ \omega_\psi(g) (\varphi_1\otimes \varphi_2) (x_0,y_0) \\&= \psi(\left\langle y_0,b^*x_0 \right\rangle) \omega_\psi(g) (\varphi_1\otimes \varphi_2) (x_0,y_0). \end{align*} $$

Since $\overline {\delta } = -\delta $ and $\left \langle f_{-n},f_n \right \rangle _W = -1$ , we have

$$ \begin{align*} \psi(\left\langle y_0,b^*x_0 \right\rangle) &= \psi_E\left( \sum_{i=1}^n \left\langle te_0,b^* \delta^{-1} e_{-i} \right\rangle_V \left\langle f_{-n}, f_{n+i-1} \right\rangle_W \right) \\&= \psi_E( \delta^{-1} t \left\langle be_0,e_{-1} \right\rangle_V) = \psi_{E,t}^\delta(\mathbf{n}_1(b)). \end{align*} $$

Hence, $\mathbf {n}_1(b)$ acts on $F_{\varphi _1 \otimes \varphi _2}$ by $\psi _{E,t}^\delta $ .

Define a map

$$\begin{align*}\mathrm{ind}_{N_{2n,S}}^{\mathrm{U}(W)}(\mu) \rightarrow \mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \end{align*}$$

by

$$\begin{align*}F \mapsto \widetilde{F}(g) = \int_{N_{2n,S} \backslash N_{2n}'}F(ug) \mu(u)^{-1} du. \end{align*}$$

Then by the same argument as in [Reference Mao and Rallis18, Proposition 2.3], one can prove that the map $\varphi _1 \otimes \varphi _2 \mapsto \widetilde {F}_{\varphi _1 \otimes \varphi _2}$ realizes an isomorphism between the maximal quotient of $\omega _\psi $ on which $N_{2n+1}$ acts by $\psi _{E,t}^\delta $ and $\mathrm {ind}_{N_{2n}'}^{\mathrm {U}(W)}(\mu )$ .

5.7 Proof of Lemma 5.7

In this subsection, we prove Lemma 5.7. To do this, we relate two models of the Weil representation.

Let $\varphi _1^0 \in \mathcal {S}(X^* \otimes W)$ and $\varphi _2^0 \in \mathcal {S}(V_0 \otimes Y^*)$ be the characteristic functions of

$$\begin{align*}\left(\bigoplus_{i=1}^n \mathfrak{o}_Ee_{-i} \right) \otimes \left( \bigoplus_{i=1}^n \mathfrak{o}_Ef_{i} \oplus \bigoplus_{i=1}^n \mathfrak{o}_Ef_{-i} \right), \quad \mathfrak{o}_E e_0 \otimes \left( \bigoplus_{i=1}^n \mathfrak{o}_Ef_{-i} \right), \end{align*}$$

respectively. Then the action $\rho = \rho _\psi $ of the Heisenberg group $H(\mathbb {W})$ on $\varphi _1^0 \otimes \varphi _2^0$ satisfies that

$$\begin{align*}\rho(a,t)(\varphi_1^0 \otimes \varphi_2^0) = \psi(t) \cdot \varphi_1^0 \otimes \varphi_2^0 \end{align*}$$

for $(a,t) \in A \oplus F$ . Moreover, the lattice model $\mathcal {S}(A)$ and the mixed model $\mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ are related by the isomorphism

$$ \begin{align*} \mathcal{S}(A) &\xrightarrow{\sim} \mathcal{S}(X^* \otimes W) \otimes \mathcal{S}(V_0 \otimes Y^*),\\ \phi &\mapsto \int_{(A \oplus F) \backslash H(\mathbb{W})} \phi(h) \rho(h)^{-1}(\varphi_1^0 \otimes \varphi_2^0)(x,y) dh. \end{align*} $$

In particular, $\phi _{t,f_{-n}} \in \mathcal {S}(A)$ corresponds to

$$\begin{align*}\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)(x,y) = \varphi_1^0(x) \cdot \rho_\psi^0(t e_0 \otimes f_{-n},0)^{-1} \varphi_2^0(y) \end{align*}$$

in $\mathcal {S}(X^* \otimes W) \otimes \mathcal {S}(V_0 \otimes Y^*)$ since $\mathrm {Supp}(\phi _{t,f_{-n}}) = (A + te_0 \otimes f_{-n}) \oplus F$ . Therefore, under the map

$$\begin{align*}\mathcal{S}(A) \rightarrow \mathrm{ind}_{N_{2n}'}^{\mathrm{U}(W)}(\mu) \end{align*}$$

obtained above, the image of $\phi _{t,f_{-n}}$ is $\widetilde {F}_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}$ .

Now we prove Lemma 5.7.

Proof of Lemma 5.7.

First, we consider $F_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}$ . Note that it is left $N_{2n,S}$ -invariant and right ${}^tK_{2m}^W$ -invariant. We claim that if

$$\begin{align*}F_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(g) \not= 0, \end{align*}$$

then

$$\begin{align*}g \in N_{2n,S} \cdot \mathbf{m}_Y(a) \cdot {}^tK_{2m}^W \end{align*}$$

for some $a \in \mathrm {GL}(Y) \cong \mathrm {GL}_n(E)$ such that $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ and

$$\begin{align*}a^*f_{-n}-f_{-n} \in \bigoplus_{i=1}^n \mathfrak{p}_E^m f_{-i}. \end{align*}$$

By the Iwasawa decomposition, we have $\mathrm {U}(W) = Q_{2n} K_0^W$ . Let $K_S$ and $K_M$ be subgroups of $K_0^W$ defined by

By the Bruhat decomposition for a finite unitary group over $\mathfrak {o}_F/\mathfrak {p}_F$ , we have

$$ \begin{align*} K_0^W &= K_S K_M \cup K_S J_{2n}^{-1} K_M \\&= (K_S \cap Q_{2n}) K_M \cup (K_S \cap Q_{2n}) J_{2n}^{-1} K_M. \end{align*} $$

Since $J_{2n} \in K_0^W$ and $J_{2n}^{-1} K_M J_{2n} = {}^t K_M$ , by the multiplication of $J_{2n}$ from the right, we have

$$\begin{align*}K_0^W = (K_S \cap Q_{2n}) J_{2n} {}^tK_M \cup (K_S \cap Q_{2n}) {}^tK_M. \end{align*}$$

Hence,

$$ \begin{align*} \mathrm{U}(W) &= Q_{2n} J_{2n} {}^tK_M \cup Q_{2n} {}^tK_M \\&= N_{2n,S}M_{2n,S} J_{2n} {}^tK_M \cup N_{2n,S}M_{2n,S} {}^tK_M. \end{align*} $$

Therefore, we may assume that $g = \mathbf {m}_Y(a)J_{2n}k$ or $g = \mathbf {m}_Y(a)k$ for some $a \in \mathrm {GL}(Y)$ and $k \in {}^tK_M$ .

Assume that $g = \mathbf {m}_Y(a)J_{2n}k$ is in the former case. Since $\varphi _1^0$ and $\varphi _2^0$ are fixed by $K_{0}^W$ , and since $\omega _\psi ^0(g) \circ \rho _\psi ^0(h) \circ \omega _\psi ^0(g)^{-1} = \rho _\psi ^0(gh)$ , we have

$$ \begin{align*} F_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(g) &= \omega_\psi(g) \circ \rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)(x_0,y_0) \\&= \varphi_1(g^{-1}x_0) \cdot \omega_\psi^0(g) \rho_\psi^0(t e_0 \otimes f_{-n},0)^{-1}\varphi_2^0(y_0) \\&= \varphi_1(a^{-1}x_0) \cdot \omega_\psi^0(\mathbf{m}_Y(a)) \rho_\psi^0(t e_0 \otimes J_{2n} k f_{-n},0)^{-1} \varphi_2^0(y_0). \end{align*} $$

Note that $\varphi _1(a^{-1}x_0) \not = 0$ if and only if $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ . However, since $k \in {}^tK_M$ , if we write $J_{2n} k f_{-n} = y+y^*$ with $y \in Y$ and $y^* \in Y^*$ , then $y^* \in \oplus _{i=1}^n \mathfrak {p}_E f_{-i}$ . Up to a nonzero constant, $\omega _\psi ^0(\mathbf {m}_Y(a)) \rho _\psi ^0(t e_0 \otimes J_{2n} k f_{-n},0)^{-1} \varphi _2^0(y_0)$ is equal to

$$\begin{align*}\varphi_2^0( te_0 \otimes (a^*f_{-n} - y^*) ). \end{align*}$$

If this is nonzero, then we must have $t(a^*f_{-n} - y^*) \in \oplus _{i=1}^n \mathfrak {o}_E f_{-i}$ . When $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ , this implies that

$$\begin{align*}f_{-n} \in (a^*)^{-1}y^* + \bigoplus_{i=1}^n \mathfrak{p}_E^m f_{-i} \subset \bigoplus_{i=1}^n \mathfrak{p}_E f_{-i}. \end{align*}$$

This is impossible. Hence, we have $F_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}(g) = 0$ .

Next, we assume that $g = \mathbf {m}_Y(a)k$ is in the latter case. By the Iwahori decomposition, we may further assume that $kf_{-n} = f_{-n}$ . Then

$$ \begin{align*} F_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(g) &= \omega_\psi(g) \circ \rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)(x_0,y_0) \\&= \varphi_1(g^{-1}x_0) \cdot \omega_\psi^0(g) \rho_\psi^0(t e_0 \otimes f_{-n})^{-1}\varphi_2^0(y_0) \\&= \varphi_1(a^{-1}x_0) \cdot \omega_\psi^0(\mathbf{m}_Y(a)) \rho_\psi^0(t e_0 \otimes kf_{-n})^{-1} \varphi_2^0(y_0) \\&= \varphi_1(a^{-1}x_0) \cdot \omega_\psi^0(\mathbf{m}_Y(a)) \rho_\psi^0(t e_0 \otimes f_{-n})^{-1} \varphi_2^0(y_0). \end{align*} $$

Up to a nonzero constant, it is equal to

$$\begin{align*}\varphi_1(a^{-1}x_0) \cdot \varphi_2^0( te_0 \otimes (a^*f_{-n}-f_{-n}) ). \end{align*}$$

If this is nonzero, then $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ and

$$\begin{align*}a^*f_{-n}-f_{-n} \in \bigoplus_{i=1}^n \mathfrak{p}_E^m f_{-i}. \end{align*}$$

This proves the claim.

Now we consider $\widetilde {F}_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}$ . Note that it is left $N_{2n}'$ -invariant and right ${}^tK_{2m}^W$ -invariant. Suppose that $\widetilde {F}_{\rho (t e_0 \otimes f_{-n},0)^{-1} (\varphi _1^0 \otimes \varphi _2^0)}(g) \not = 0$ . By the claim, we may assume that $g = \mathbf {m}_Y(a)$ with $a \in \mathrm {GL}(Y)$ satisfying the conditions in the claim. By the Iwasawa decomposition, we may further assume that $a = a_d a_0$ such that

• • $\left \langle a_d f_i, f_{-j} \right \rangle _W = \varpi ^{\lambda _i} \delta _{i,j}$ for some $\lambda _i \in \mathbb {Z}$ ;

• • $a_0 \in \mathrm {GL}_n(\mathfrak {o}_E)$ via $\mathrm {GL}(Y) \cong \mathrm {GL}_n(E)$ .

Since $a^{-1} \in \mathrm {M}_n(\mathfrak {o}_E)$ , we have $\lambda _i \leq 0$ for $1 \leq i \leq n$ . Note that

$$\begin{align*}a^* f_{-n}-f_{-n} = a_0^* a_d^* f_{-n}-f_{-n} = a_0^* \varpi^{\lambda_n}f_{-n}-f_{-n}. \end{align*}$$

Since this is in $\oplus _{i=1}^n \mathfrak {p}_E^m f_{-i}$ , we have $\lambda _n = 0$ and $\mathbf {m}_Y(a_0) \in {}^tK_{2m}^W$ . Hence, we may assume that $a_0 = \mathbf {1}_X$ (i.e., $g = \mathbf {m}_Y(a_d)$ ). For $2 \leq i \leq n$ and $x \in \mathfrak {o}_E$ , define $u_i \in N_{2n}'$ so that

$$\begin{align*}u_i f_j - f_j = \left\{ \begin{aligned} &x \cdot f_{i-1} &\quad&\text{if } j=i, \\ &0 &\quad&\text{if } j\not=i. \end{aligned} \right. \end{align*}$$

Then $u_i \in {}^tK_{2m}^W$ . Hence,

$$ \begin{align*} 0 &\not= \widetilde{F}_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(\mathbf{m}_Y(a_d)) \\&= \widetilde{F}_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(\mathbf{m}_Y(a_d)u_i) \\&= \mu(\mathbf{m}_Y(a_d) u_i \mathbf{m}_Y(a_d)^{-1}) \widetilde{F}_{\rho(t e_0 \otimes f_{-n},0)^{-1} (\varphi_1^0 \otimes \varphi_2^0)}(\mathbf{m}_Y(a_d)) \end{align*} $$

so that $\mu (\mathbf {m}_Y(a_d) u_i \mathbf {m}_Y(a_d)^{-1}) = 1$ for any $x \in \mathfrak {o}_E$ . Note that

$$ \begin{align*} \mu(\mathbf{m}_Y(a_d) u_i \mathbf{m}_Y(a_d)^{-1}) &= \psi_E( \left\langle \mathbf{m}_Y(a_d) u_i \mathbf{m}_Y(a_d)^{-1} f_{i}, f_{-i+1} \right\rangle ) \\&= \psi_E( \varpi^{\lambda_{i-1}-\lambda_i}x). \end{align*} $$

Hence, $\psi _E( \varpi ^{\lambda _{i-1}-\lambda _i}x) = 1$ for any $x \in \mathfrak {o}_E$ . This implies that $\lambda _{i-1} \geq \lambda _i$ . In conclusion, we have

$$\begin{align*}0 \geq \lambda_1 \geq \dots \geq \lambda_{n-1} \geq \lambda_n = 0 \end{align*}$$

so that $\lambda _1=\dots =\lambda _n=0$ . This means that $a_d = \mathbf {1}_X$ . This completes the proof of Lemma 5.7.