Proof. It follows from [Reference Rapoport, Smithling and ZhangRZ96] that $\mathcal {N}$ is a separated formal scheme over $\operatorname {\mathrm {Spf}} O_{\breve {E}}$. The formal smoothness of $\mathcal {N}$ follow from the smoothness of its local model, which is [Reference Rapoport, Smithling and ZhangRSZ17, Proposition 3.10], and the dimension also follows. For the last assertion, our moduli functor $\mathcal {N}$ is the disjoint union of $\mathcal {N}_{(0,0)}$ and $\mathcal {N}_{(0,1)}$ from [Reference WuWu, Section 3.4], each of which is connected by [Reference WuWu, Theorem 5.18(2)].Footnote ⁸

Proof. Part (1) follows from [Reference JacobowitzJac62, Propositions 4.3 & 8.1]. Part (2) follows from the canonicity of the Jordan splitting on [Reference JacobowitzJac62, Page 449]. Part (3) follows from a direct calculation of $L^\vee $.

Proof. Put $V := L\otimes _{O_F}F$. For an integral $O_E$-lattice $L'$ of V, we equip the k-vector space $L^{\prime }_k := L'\otimes _{O_E}O_E/(u)$ with a k-valued pairing $\langle \,,\,\rangle _{L^{\prime }_k}$ by the formula

$$ \begin{align*} \langle x,y\rangle_{L^{\prime}_k} := u\cdot (x^\sharp,y^\sharp)_V\mod(u) \end{align*} $$

where $x^\sharp $ and $y^\sharp $ are arbitrary lifts of x and y, respectively. Then $L^{\prime }_k$ becomes a symplectic space over k of dimension m whose radical has dimension $t(L')$. Similarly, we have $H_{s,k}$, which is a nondegenerate symplectic space over k of dimension $2s$. We denote by $\operatorname {\mathrm {Isom}}_{L^{\prime }_k,H_{s,k}}$ the k-scheme of isometries from $L^{\prime }_k$ to $H_{s,k}$.

By the same argument in [Reference Cho and YamauchiCY20, Section 3.3], we have

$$ \begin{align*} \mathrm{Den}(H_s,L)=q^{-m(4s-m+1)/2}\cdot\sum_{L\subseteq L'\subseteq L^{\prime \vee}}|L'/L|^{m-2s}\left|\operatorname{\mathrm{Isom}}_{L^{\prime}_k,H_{s,k}}(k)\right|. \end{align*} $$

Thus, it remains to show that

(2.2)$$ \begin{align} \left|\operatorname{\mathrm{Isom}}_{L^{\prime}_k,H_{s,k}}(k)\right|=q^{m(4s-m+1)/2}\prod_{s-\frac{m+t(L')}{2}<i\leqslant s}(1-q^{-2i}). \end{align} $$

We fix a decomposition $L^{\prime }_k=L_0\oplus L_1$ in which $L_0$ is nondegenerate and $L_1$ is the radical of $L^{\prime }_k$. We have a morphism $\pi \colon \operatorname {\mathrm {Isom}}_{L^{\prime }_k,H_{s,k}}\to \operatorname {\mathrm {Isom}}_{L_0,H_{s,k}}$ given by restriction, such that for every element $f\in \operatorname {\mathrm {Isom}}_{L_0,H_{s,k}}(k)$, the fibre $\pi ^{-1}f$ is isomorphic to $\operatorname {\mathrm {Isom}}_{L_1,\operatorname {\mathrm {im}}(f)^\perp }$. As $\operatorname {\mathrm {im}}(f)^\perp $ is isomorphic to $H_{s-\frac {m-t(L')}{2},k}$, it suffices to show (2.2) in the two extremal cases: $t(L')=0$ and $t(L')=m$.

Suppose that $t(L')=0$; that is, $L^{\prime }_k$ is nondegenerate. Note that $\operatorname {\mathrm {Sp}}(H_{s,k})$ acts on $\operatorname {\mathrm {Isom}}_{L^{\prime }_k,H_{s,k}}$ transitively, with the stabiliser isomorphic to $\operatorname {\mathrm {Sp}}(H_{s-\frac {m}{2},k})$. Thus, we have

$$ \begin{align*} \left|\operatorname{\mathrm{Isom}}_{L^{\prime}_k,H_{s,k}}(k)\right|&=\frac{|\operatorname{\mathrm{Sp}}(H_{s,k})(k)|}{\left|\operatorname{\mathrm{Sp}}(H_{s-\frac{m}{2},k})(k)\right|} \\ &=\frac{q^{s^2}\prod_{i=1}^s(q^{2i}-1)}{q^{\left(s-\frac{m}{2}\right)^2}\prod_{i=1}^{s-\frac{m}{2}}(q^{2i}-1)} \\ &=q^{m(4s-m+1)/2}\prod_{s-\frac{m}{2}<i\leqslant s}(1-q^{-2i}), \end{align*} $$

which confirms (2.2).

Suppose that $t(L')=m$; that is, $L^{\prime }_k$ is isotropic. Note that $\operatorname {\mathrm {Sp}}(H_{s,k})$ acts on $\operatorname {\mathrm {Isom}}_{L^{\prime }_k,H_{s,k}}$ transitively, with the stabiliser Q fitting into a short exact sequence

$$ \begin{align*} 1 \to U_m \to Q \to \operatorname{\mathrm{Sp}}(H_{s-m,k}) \to 1 \end{align*} $$

in which $U_m$ is a unipotent subgroup of $\operatorname {\mathrm {Sp}}(H_{s,k})$ of Levi type $\operatorname {\mathrm {GL}}_{m,k}\times ^(H_{s-m,k})$. Thus, we have

$$ \begin{align*} |\operatorname{\mathrm{Isom}}_{L^{\prime}_k,H_{s,k}}(k)| &=\frac{|\operatorname{\mathrm{Sp}}(H_{s,k})(k)|}{|U_m(k)|\cdot|\operatorname{\mathrm{Sp}}(H_{s-m,k})(k)|} \\ &=\frac{q^{s^2}\prod_{i=1}^s(q^{2i}-1)}{q^{m(2s-2m)+\frac{m(m+1)}{2}}\cdot q^{\left(s-m\right)^2}\prod_{i=1}^{s-m}(q^{2i}-1)} \\ &=q^{m(4s-m+1)/2}\prod_{s-m<i\leqslant s}(1-q^{-2i}), \end{align*} $$

which confirms (2.2).

Thus, (2.2) is proved and the lemma follows.

Proof. The identity (2.3) is a direct consequence of Lemma 2.15 and Definition 2.16. The identity (2.4) is a consequence of (2.3).

Proof. Part (1) is simply the well-known method of computing the Smith normal form of $uT$ (over $O_E$) using ideals generated by determinants of minors. For (2), take a normal basis $\{x_1,\dots ,x_{m-1}\}$ of L (Definition 2.11) such that $\langle x_1,\dots ,x_{m-1-t(L')}\rangle $ is self-dual. Applying (1) to the basis $\{x_1,\dots ,x_{m-1},x\}$ of L, we know that $t(L)=t(L')+1$ if $(x_i,x)_L\in O_E$ for every $m-t(L')\leqslant i\leqslant {m-1}$; otherwise, we have $t(L)=t(L')-1$. In particular, (2) follows.

Proof. For (1), the inverse map $\mathfrak {E}\to \mathfrak {L}$ is the one that sends $(L^{\flat \prime },\delta ,\varepsilon )$ to the $O_E$-lattice L generated by $L^{\flat \prime }$ and $\varepsilon _L(x)+x$ for every $x\in u^{\delta _L}(V_{L^\flat }^\perp )^{\mathrm {int}}$. The rest of (1) is straightforward.

Part (2) is simply Lemma 2.23(2).

Part (4) follows by applying (3) to generators z of $O_E$-modules $u^{-1}(V_{L^\flat }^\perp )^{\mathrm {int}}$ and $u^{-2}(V_{L^\flat }^\perp )^{\mathrm {int}}$ and then taking the difference.

Now we prove (3), which is the most difficult one. For every $x\in \boldsymbol {V}$, we denote by $x'\in V_{L^\flat }$ the first component of x with respect to the orthogonal decomposition $\boldsymbol {V}=V_{L^\flat }\oplus V_{L^\flat }^\perp $. Put

$$ \begin{align*} \Omega := \{x\in\boldsymbol{V}^{\mathrm{int}}\mathbin{|} x'\in(L^{\flat\prime})^\vee\},\qquad \Omega^\circ := \{x\in\boldsymbol{V}^{\mathrm{int}}\mathbin{|} x'\in u(L^{\flat\prime})^\vee+L^{\flat\prime}\}. \end{align*} $$

Note that both $\Omega $ and $\Omega ^\circ $ are open compact subsets of $\boldsymbol {V}$ stable under the translation by $L^{\prime \flat }$. For an element $L\in \mathfrak {L}$ corresponding to $(L^{\flat \prime },\delta ,\varepsilon )\in \mathfrak {E}$ from (1), L is integral if and only $\varepsilon (x)+x\subseteq \Omega $ for every $x\in u^\delta (V_{L^\flat }^\perp )^{\mathrm {int}}$. By (2), for such L,

$$ \begin{align*} t(L)= \begin{cases} t(L^{\flat\prime})+1, &\text{if } \varepsilon(x)+x\subseteq\Omega^\circ \text{ for every } x\in u^\delta(V_{L^\flat}^\perp)^{\mathrm{int}}\setminus u^{\delta+1}(V_{L^\flat}^\perp)^{\mathrm{int}}, \\ t(L^{\flat\prime})-1, &\text{if } \varepsilon(x)+x\subseteq\Omega\setminus\Omega^\circ \text{ for every } x\in u^\delta(V_{L^\flat}^\perp)^{\mathrm{int}}\setminus u^{\delta+1}(V_{L^\flat}^\perp)^{\mathrm{int}}. \end{cases} \end{align*} $$

Thus, we may replace the term corresponding to L in the summation in (3) by an integration over the region $\bigcup _{x\in u^\delta (V_{L^\flat }^\perp )^{\mathrm {int}}\setminus u^{\delta +1}(V_{L^\flat }^\perp )^{\mathrm {int}}}(\varepsilon (x)+x)$ of $\Omega $. It follows that

$$ \begin{align*} \sum_{\substack{L\subseteq L^\vee \\ L\cap V_{L^\flat}=L^{\flat\prime}}}q^{-\delta_L}|\mu(t(L))| =\frac{1}{C}\left(\int_{\Omega^\circ\setminus V_{L^\flat}}|\mu(t(L^{\flat\prime})+1)|\,\mathrm{d} x+\int_{\Omega\setminus(\Omega^\circ\cup V_{L^\flat})}|\mu(t(L^{\flat\prime})-1)|\,\mathrm{d} x\right), \end{align*} $$

which is convergent, where

$$ \begin{align*} C=\operatorname{\mathrm{vol}}(L^{\flat\prime})\cdot\operatorname{\mathrm{vol}}((V_{L^\flat}^\perp)^{\mathrm{int}}\setminus u(V_{L^\flat}^\perp)^{\mathrm{int}}). \end{align*} $$

Now we take an element $z\in \boldsymbol {V}\setminus \boldsymbol {V}^{\mathrm {int}}$. We may assume $z'\in (L^{\flat \prime })^\vee $ since otherwise the summation in (3) is empty. Put

$$ \begin{align*} \Omega_z := \{x\in\Omega\mathbin{|}(x,z)_{\boldsymbol{V}}\in u^{-1}O_E\},\qquad \Omega^\circ_z := \{x\in\Omega^\circ\mathbin{|}(x,z)_{\boldsymbol{V}}\in u^{-1}O_E\}, \end{align*} $$

both stable under the translation by $L^{\prime \flat }$ since $z'\in (L^{\flat \prime })^\vee $. Similarly, we have

$$ \begin{align*} \sum_{\substack{L\subseteq L^\vee \\ L\cap V_{L^\flat}=L^{\flat\prime}\\ z\in L^\vee}}q^{-\delta_L}\mu(t(L)) &=\frac{1}{C}\left(\int_{\Omega_z^\circ\setminus V_{L^\flat}}\mu(t(L^{\flat\prime})+1)\,\mathrm{d} x+\int_{\Omega_z\setminus(\Omega_z^\circ\cup V_{L^\flat})}\mu(t(L^{\flat\prime})-1)\,\mathrm{d} x\right) \\ &=\frac{\mu(t(L^{\flat\prime})-1)}{C}\left(\operatorname{\mathrm{vol}}(\Omega_z\setminus\Omega_z^\circ)+\left(1-q^{t(L^{\flat\prime})-1}\right)\operatorname{\mathrm{vol}}(\Omega_z^\circ)\right) \\ &=\frac{\mu(t(L^{\flat\prime})-1)}{C}\left(\operatorname{\mathrm{vol}}(\Omega_z)-q^{t(L^{\flat\prime})-1}\operatorname{\mathrm{vol}}(\Omega_z^\circ)\right), \end{align*} $$

where we have used $t(L^{\flat \prime })>1$ in the second equality. Thus, it remains to show that

(2.5)$$ \begin{align} \operatorname{\mathrm{vol}}(\Omega_z)=q^{t(L^{\flat\prime})-1}\operatorname{\mathrm{vol}}(\Omega_z^\circ). \end{align} $$

We fix an orthogonal decomposition $L^{\flat \prime }=L_0\oplus L_1$ in which $L_0$ is self-dual and $L_1$ is of both rank and type $t(L^{\flat \prime })$. Since both $\Omega _z$ and $\Omega _z^\circ $ depend only on the coset $z+L^{\flat \prime }$, we may assume $z'\in L_1^\vee $ and anisotropic. Let $V_2\subseteq \boldsymbol {V}$ be the orthogonal complement of $L_0+\langle z\rangle $. We claim

• (*) There exists an integral $O_E$-lattice $L_2$ of $V_2$ of of type $t(L^{\flat \prime })$ such that

  (2.6)$$ \begin{align} (u^i L_2^\vee)^{\mathrm{int}}=\{x\in V_2^{\mathrm{int}}\mathbin{|} x'\in u^iL_1^\vee\} \end{align} $$
  for $i=0,1$.

Assuming ($*$), by construction, we have

$$ \begin{align*} \{x\in\boldsymbol{V}\mathbin{|}(x,z)_{\boldsymbol{V}}\in u^{-1}O_E\}=L_0\otimes_{O_F}F\oplus\langle z\rangle^\vee\oplus V_2. \end{align*} $$

Now we use the condition $z\not \in \boldsymbol {V}^{\mathrm {int}}$, which implies that $\langle z\rangle ^\vee \subseteq u\langle z\rangle \cap \boldsymbol {V}^{\mathrm {int}}$. Combining with (2.6), we obtain

$$ \begin{align*} \Omega_z=L_0\times\langle z\rangle^\vee\times(L_2^\vee)^{\mathrm{int}},\qquad \Omega_z^\circ=L_0\times\langle z\rangle^\vee\times(uL_2^\vee)^{\mathrm{int}}. \end{align*} $$

Thus, (2.5) follows from Lemma 2.25. Part (3) is proved.

Now we show ($*$). There are two cases.

First, we assume $z\neq z'$; that is, $z\not \in V_{L^\flat }$. Let $L_2$ be the unique $O_E$-lattice of $V_2$ satisfying

(2.7)$$ \begin{align} L_2^\vee=\{x\in V_2\mathbin{|} x'\in L_1^\vee\}. \end{align} $$

Then (2.6) clearly holds. Thus, it remains to show that $L_2$ is integral of type $t(L^{\flat \prime })$. Put $w := z-z'\in V_{L^\flat }^\perp $, which is nonzero and hence anisotropic. Then

$$ \begin{align*} \bar{z} := z'-\frac{(z',z')_{\boldsymbol{V}}}{(w,w)_{\boldsymbol{V}}}w \end{align*} $$

is the unique element in $V_2$ such that $\bar {z}'=z'$. To compute $L_2$, we write

$$ \begin{align*} L_1^\vee=M+\langle y+\alpha z'\rangle \end{align*} $$

for some $y\in V_{L^\flat }\cap V_2$ and $\alpha \in E\setminus uO_E$, where $M := L_1^\vee \cap V_2$. Then

$$ \begin{align*} M^\dagger := L_1\cap V_2=\{x\in M^\vee\mathbin{|} (x,y)_{\boldsymbol{V}}\in u^{-1}O_E\}. \end{align*} $$

Since $M^\vee /M^\dagger $ is isomorphic to an $O_E$-submodule of $E/u^{-1}O_E$, we may take an element $y^\dagger \in M^\vee $ that generates $M^\vee /M^\dagger $. Then we have

$$ \begin{align*} L_1=M^\dagger+\langle y^\dagger+\alpha^\dagger z'\rangle \end{align*} $$

for some $\alpha ^\dagger \in E^\times $ such that $(y^\dagger ,y)_{\boldsymbol {V}}+\alpha ^\dagger \alpha ^{\mathtt {c}}(z',z')_{\boldsymbol {V}}\in u^{-1}O_E$. Now by (2.7), we have

$$ \begin{align*} L_2^\vee=M+\langle y+\alpha\bar{z}\rangle. \end{align*} $$

By the same argument, we have

$$ \begin{align*} L_2=M^\dagger+\langle y^\dagger+\alpha^\dagger\rho\bar{z}\rangle, \end{align*} $$

where

$$ \begin{align*} \rho := \frac{(z',z')_{\boldsymbol{V}}}{(\bar{z},\bar{z})_{\boldsymbol{V}}}. \end{align*} $$

By Lemma 2.23(2), we have $t(L_2)=t(L_1)=t(L^{\flat \prime })$ as long as $L_2$ is integral. Thus, it suffices to show that $y^\dagger +\alpha ^\dagger \rho \bar {z}\in \boldsymbol {V}^{\mathrm {int}}$. We compute

$$ \begin{align*} &(y^\dagger+\alpha^\dagger\rho\bar{z},y^\dagger+\alpha^\dagger\rho\bar{z})_{\boldsymbol{V}}-(y^\dagger+\alpha^\dagger z',y^\dagger+\alpha^\dagger z')_{\boldsymbol{V}} \\ &=(\alpha^\dagger\rho\bar{z},\alpha^\dagger\rho\bar{z})_{\boldsymbol{V}}-(\alpha^\dagger z',\alpha^\dagger z')_{\boldsymbol{V}} \\ &=\operatorname{\mathrm{Nm}}_{E/F}(\alpha^\dagger)\left(\frac{(z',z')_{\boldsymbol{V}}^2}{(\bar{z},\bar{z})_{\boldsymbol{V}}}-(z',z')_{\boldsymbol{V}}\right) \\ &=\operatorname{\mathrm{Nm}}_{E/F}(\alpha^\dagger)(z',z')_{\boldsymbol{V}}\left(\frac{(z',z')_{\boldsymbol{V}}}{(z',z')_{\boldsymbol{V}}+\frac{(z',z')_{\boldsymbol{V}}^2}{(w,w)_{\boldsymbol{V}}}}-1\right) \\ &=\operatorname{\mathrm{Nm}}_{E/F}(\alpha^\dagger)(z',z')_{\boldsymbol{V}}\left(\frac{(w,w)_{\boldsymbol{V}}}{(z',z')_{\boldsymbol{V}}+(w,w)_{\boldsymbol{V}}}-1\right) \\ &=\frac{-(\alpha^\dagger)^{\mathtt{c}}}{\alpha^\dagger}\frac{(\alpha^\dagger z',z')_{\boldsymbol{V}}^2}{(z,z)_{\boldsymbol{V}}}. \end{align*} $$

As $z'\in L_1^\vee $, we have $(\alpha ^\dagger z',z')_{\boldsymbol {V}}\in u^{-1}O_E$. As $z\not \in \boldsymbol {V}^{\mathrm {int}}$, we have $(z,z)_{\boldsymbol {V}}\not \in u^{-1}O_E$. Together, we have $\frac {(\alpha ^\dagger z',z')_{\boldsymbol {V}}^2}{(z,z)_{\boldsymbol {V}}}\in O_F$. Thus, $y^\dagger +\alpha ^\dagger \rho \bar {z}\in \boldsymbol {V}^{\mathrm {int}}$ as $y^\dagger +\alpha ^\dagger z'\in \boldsymbol {V}^{\mathrm {int}}$; hence, $L_2$ meets the requirement in ($*$).

Second, we assume $z=z'$; that is, $z\in V_{L^\flat }$. Take $L_2=(L_1^\vee \cap V_2)^\vee \oplus u^\delta (V_{L^\flat }^\perp )^{\mathrm {int}}$ for some integer $\delta \geqslant 0$ determined later. We show that $(L_1^\vee \cap V_2)^\vee $ is an integral hermitian $O_E$-module of type $t(L^{\flat \prime })-1$. As in the previous case, we write

$$ \begin{align*} L_1^\vee=M+\langle y+\alpha z'\rangle \end{align*} $$

for some $y\in V_{L^\flat }\cap V_2$ and $\alpha \in E\setminus uO_E$, where $M := L_1^\vee \cap V_2$. Then

$$ \begin{align*} L_1=M^\dagger+\langle y^\dagger+\alpha^\dagger z'\rangle, \end{align*} $$

so that $M^\vee $ is generated by $M^\dagger $ and $y^\dagger $. As $L_1$ is of type $t(L^{\flat \prime })$, which is its rank, we have $L_1\subseteq uL_1^\vee $; that is,

$$ \begin{align*} M^\dagger+\langle y^\dagger+\alpha^\dagger z'\rangle\subseteq uM+u\langle y+\alpha z'\rangle; \end{align*} $$

hence, $M^\dagger \subseteq uM$. As $z'\in L_1^\vee $, we have $(\alpha z',z')_{\boldsymbol {V}}\in u^{-1}O_E$. As $z'=z\not \in \boldsymbol {V}^{\mathrm {int}}$, we have $(z',z')_{\boldsymbol {V}}\not \in u^{-1}O_E$; hence, $\alpha ^\dagger \in uO_E$. Again, as $z'\in L_1^\vee $, we have $\alpha ^\dagger z'\in uL_1^\vee $; hence, $y^\dagger \in uL_1^\vee \cap V_2= uM$. Together, we obtain $M^\vee \subseteq uM$; that is, $(L_1^\vee \cap V_2)^\vee $ is an integral hermitian $O_E$-module of type $t(L^{\flat \prime })-1$.

Consequently, $L_2$ is an integral $O_E$-lattice of $V_2$ of type $t(L^{\flat \prime })$. Since $L_2^\vee =(L_1^\vee \cap V_2)\oplus u^{-\delta -1}(V_{L^\flat }^\perp )^{\mathrm {int}}$, it is clear that for $\delta $ sufficiently large, (2.6) holds for $i=0,1$. Thus, ($*$) is proved.

The lemma is all proved.

Proof. Put $V := L\otimes _{O_F}F$. We prove by induction on $\operatorname {\mathrm {val}}(L)$ for integral $O_E$-lattices L of V with $t(L)=2m+1$ that (2.8) holds.

The initial case is such that $\operatorname {\mathrm {val}}(L)=2m+1$; that is, $L^\vee =u^{-1}L$. The pairing $u^2(\,,\,)_V$ induces a nondegenerate quadratic form on $L^\vee /L$. It is clear that $(L^\vee )^{\mathrm {int}}/L$ is exactly the set of isotropic vectors in $L^\vee /L$ under the previous form. In particular, we have

$$ \begin{align*} \left|(L^\vee)^{\mathrm{int}}/L\right|=q^{2m}=q^{2m}\cdot\left|(uL^\vee)^{\mathrm{int}}/L\right|. \end{align*} $$

Now we consider L with $\operatorname {\mathrm {val}}(L)>2m+1$ and suppose that (2.8) holds for such $L'$ with $\operatorname {\mathrm {val}}(L')<\operatorname {\mathrm {val}}(L)$. Choose an orthogonal decomposition $L=L_0\oplus L_1$ in which $L_0$ is an integral hermitian $O_E$-module with fundamental invariants $(1,\dots ,1)$ and such that all fundamental invariants of $L_1$ are at least $2$. In particular, $L_1$ has positive rank. It is easy to see that we may choose a hermitian $O_E$-module $L^{\prime }_1$ contained in $u^{-1}L_1$ satisfying $L_1\varsubsetneq L^{\prime }_1$ and $t(L^{\prime }_1)=t(L_1)$. Put $L' := L_0\oplus L^{\prime }_1$. By the induction hypothesis, we have

$$ \begin{align*} \left|(L^{\prime \vee})^{\mathrm{int}}/L'\right|=q^{2m}\cdot\left|(uL^{\prime \vee})^{\mathrm{int}}/L'\right|. \end{align*} $$

It remains to show that

(2.9)$$ \begin{align} \left|((L^\vee)^{\mathrm{int}}\setminus (L^{\prime \vee})^{\mathrm{int}})/L\right|=q^{2m}\cdot\left|((uL^\vee)^{\mathrm{int}}\setminus(uL^{\prime \vee})^{\mathrm{int}})/L\right|. \end{align} $$

We claim that the map

$$ \begin{align*} ((L^\vee)^{\mathrm{int}}\setminus (L^{\prime \vee})^{\mathrm{int}})/L\to((uL^\vee)^{\mathrm{int}}\setminus(uL^{\prime \vee})^{\mathrm{int}})/L \end{align*} $$

given by the multiplication by u is $q^{2m}$-to-$1$.

Take an element $x\in (uL^\vee )^{\mathrm {int}}\setminus (uL^{\prime \vee })^{\mathrm {int}}$. Its preimage is bijective to the set of elements $(y_0,y_1)\in L_0/uL_0\oplus L_1/uL_1$ such that $u^{-1}(x+(y_0,y_1))\in V^{\mathrm {int}}$, which amounts to the equation

$$ \begin{align*} (x,x)_V+\operatorname{\mathrm{Tr}}_{E/F}(x,y_0)_V+\operatorname{\mathrm{Tr}}_{E/F}(x,y_1)_V+(y_0,y_0)_V \in u^2 O_F. \end{align*} $$

Since $x\in (uL_0^\vee )\times ((uL_1^\vee )^{\mathrm {int}}\setminus (u^2L_1^\vee )^{\mathrm {int}})$, there exists $y_1\in L_1$ such that $(x,y_1)_V\in O_E^\times $. In other words, for each $y_0$, the above relation defines a nontrivial linear equation on $L_1/uL_1$. Thus, the preimage of x has cardinality $q^{2m}$. We obtain (2.9) and hence complete the induction process.

Proof of Proposition 2.22.

We fix an element $L^\flat \in \flat (\boldsymbol {V})$. If $L^\flat $ is not integral, then $\partial \mathrm {Den}_{L^\flat }^{\mathrm {v}}\equiv 0$; hence, the proposition is trivial. Thus, we now assume $L^\flat $ integral and will freely adopt notation from Lemma 2.24.

To show that $\partial \mathrm {Den}_{L^\flat }^{\mathrm {v}}$ extends to a compactly supported locally constant function on $\boldsymbol {V}$, it suffices to show that for every $y\in V_{L^\flat }/L^\flat $, there exists an integer $\delta (y)>0$ such that $\partial \mathrm {Den}_{L^\flat }^{\mathrm {v}}(y+x)$ is constant for $x\in u^{\delta (y)}(V_{L^\flat }^\perp )^{\mathrm {int}}\setminus \{0\}$. If $L^\flat +\langle y\rangle $ is not integral, then there exists $\delta (y)>0$ such that $L^\flat +\langle y+x\rangle $ is not integral for $x\in u^{\delta (y)}(V_{L^\flat }^\perp )^{\mathrm {int}}\setminus \{0\}$, which implies $\partial \mathrm {Den}_{L^\flat }^{\mathrm {v}}(y+x)=0$.

Now we fix an element $y\in V_{L^\flat }/L^\flat $ such that $L^\flat +\langle y\rangle $ is integral. We claim that we may take $\delta (y)=a_{n-1}$, which is the maximal element in the fundamental invariants of $L^\flat $. It amounts to showing that for every fixed pair $(f_1,f_2)$ of generators of the $O_E$-module $(V_{L^\flat }^\perp )^{\mathrm {int}}$, we have

(2.10)$$ \begin{align} \partial\mathrm{Den}_{L^\flat}^{\mathrm{v}}(y+u^\delta f_1)-\partial\mathrm{Den}_{L^\flat}^{\mathrm{v}}(y+u^{\delta-1} f_2)=0 \end{align} $$

for $\delta>a_{n-1}$. For every $\delta '\in \mathbb {Z}$, we define two sets

$$ \begin{align*} \mathfrak{L}_1^{\delta'}& := \{L\in\mathfrak{L}\mathbin{|} L\subseteq L^\vee,\delta_L=\delta',y+u^\delta f_1\in L\},\\ \mathfrak{L}_2^{\delta'}& := \{L\in\mathfrak{L}\mathbin{|} L\subseteq L^\vee,\delta_L=\delta',y+u^{\delta-1}f_2\in L\}. \end{align*} $$

By Remark 2.21(4), we have

$$ \begin{align*} \partial\mathrm{Den}_{L^\flat}^{\mathrm{v}}(y+u^\delta f_1)&=2\sum_{\delta'\leqslant\delta}\sum_{\substack{L\in\mathfrak{L}_1^{\delta'}\\ t(L\cap V_{L^\flat})>1}}\mu(t(L)) =2\sum_{\substack{L^\flat\subseteq L^{\flat\prime}\subseteq (L^{\flat\prime})^\vee\\t(L^{\flat\prime})>1}}\sum_{\delta'\leqslant\delta} \sum_{\substack{L\in\mathfrak{L}_1^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L));\\ \partial\mathrm{Den}_{L^\flat}^{\mathrm{v}}(y+u^{\delta-1}f_2)&=2\sum_{\delta'\leqslant\delta-1}\sum_{\substack{L\in\mathfrak{L}_2^{\delta'}\\ t(L\cap V_{L^\flat})>1}}\mu(t(L)) =2\sum_{\substack{L^\flat\subseteq L^{\flat\prime}\subseteq (L^{\flat\prime})^\vee\\t(L^{\flat\prime})>1}}\sum_{\delta'\leqslant\delta-1} \sum_{\substack{L\in\mathfrak{L}_2^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L)). \end{align*} $$

Now we claim that

(2.11)$$ \begin{align} \sum_{\delta'\leqslant\delta} \sum_{\substack{L\in\mathfrak{L}_1^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L))- \sum_{\delta'\leqslant\delta-1} \sum_{\substack{L\in\mathfrak{L}_2^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L))=0 \end{align} $$

for every $L^{\flat \prime }$ in the summation. Since $\delta>a_{n-1}$, it follows that for $\delta '<0$, we have

$$ \begin{align*} \mathfrak{L}_1^{\delta'}=\mathfrak{L}_2^{\delta'}=\{L\in\mathfrak{L}\mathbin{|} L\subseteq L^\vee,\delta_L=\delta',y\in L\}. \end{align*} $$

Thus, the left-hand side of (2.11) equals

(2.12)$$ \begin{align} \sum_{\delta'=0}^\delta \sum_{\substack{L\in\mathfrak{L}_1^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L))- \sum_{\delta'=0}^{\delta-1} \sum_{\substack{L\in\mathfrak{L}_2^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L)). \end{align} $$

However, we also have $\mathfrak {L}_1^0=\{L\in \mathfrak {L}\mathbin {|} L\subseteq L^\vee ,\delta _L=\delta ',y\in L\}$, which implies

which vanishes by Lemma 2.24(4). Thus, we obtain

(2.13)$$ \begin{align} (2.12) =\sum_{\delta'=1}^{\delta} \sum_{\substack{L\in\mathfrak{L}_1^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L))- \sum_{\delta'=0}^{\delta-1} \sum_{\substack{L\in\mathfrak{L}_2^{\delta'}\\ L\cap V_{L^\flat}=L^{\flat\prime}}}\mu(t(L)). \end{align} $$

Finally, the automorphism of $\mathfrak {E}$ sending $(L^{\flat \prime },\delta ',\varepsilon )$ to $(L^{\flat \prime },\delta '-1,\varepsilon \circ (u\alpha \cdot ))$, where $\alpha \in O_E^\times $ is the element satisfying $f_1=\alpha f_2$, induces a bijection from $\mathfrak {L}_1^{\delta '}$ to $\mathfrak {L}_2^{\delta '-1}$ preserving both $L\cap V_{L^\flat }$ and $t(L)$. Thus, (2.13) vanishes; hence, (2.11) and (2.10) hold.

Now we show that the support of $\widehat {\partial \mathrm {Den}_{L^\flat }^{\mathrm {v}}}$ is contained in $\boldsymbol {V}^{\mathrm {int}}$. Take an element $z\in \boldsymbol {V}\setminus \boldsymbol {V}^{\mathrm {int}}$. Using Remark 2.21(4), we have

which is valid and vanishes by Lemma 2.24(3).

Proposition 2.22 is proved.

Proof. This follows from [Reference WuWu, Proposition 5.13 & Theorem 5.18]. Note that we use lattices in $\boldsymbol {V}$, which is different from the hermitian space C used in [Reference WuWu], to parametrise strata. By the obvious analogue of [Reference KudlaKR11, Lemma 3.9], we may naturally identify $\boldsymbol {V}$ with C, after which the stratum $\mathcal {S}_\Lambda $ in [Reference WuWu] coincides with our stratum $\mathcal {V}_{u\Lambda ^\vee }$.

Proof. Since $\mathcal {N}(x)_{\mathrm {red}}$ is a reduced closed subscheme of $\mathcal {N}_{\mathrm {red}}$, it suffices to check that

$$ \begin{align*} \mathcal{N}(x)(\overline{k})=\bigcup_{x\in\Lambda}\mathcal{V}^\circ_\Lambda(\overline{k}). \end{align*} $$

By Definition 2.27(2), we have

$$ \begin{align*} \mathcal{N}(x)(\overline{k})\supseteq\bigcup_{x\in\Lambda}\mathcal{V}^\circ_\Lambda(\overline{k}). \end{align*} $$

For the other direction, by Proposition 2.28, we have to show that if $\Lambda $ does not contain x, then $\mathcal {N}(x)(\overline {k})\cap \mathcal {V}^\circ _\Lambda (\overline {k})=\emptyset $. Suppose that we have $s\in \mathcal {N}(x)(\overline {k})\cap \mathcal {V}^\circ _\Lambda (\overline {k})$; then s should belong to $\mathcal {V}_{\Lambda '}(\overline {k})$ where $\Lambda '$ is the $O_E$-lattice generated by $\Lambda $ and x. In particular, $\Lambda '$ is vertex and strictly contains $\Lambda $. But this contradicts with the definition of $\mathcal {V}^\circ _\Lambda $. The corollary follows.

Proof. By Corollary 2.30 and Proposition 2.28(2), it suffices to show that the intersection of all vertex $O_E$-lattices of $\boldsymbol {V}$ is $\{0\}$.

Take a nonsplit hermitian subspace $V_2$ of $\boldsymbol {V}$ of dimension $2$ and an $O_E$-lattice $L_2$ of $V_2$ of fundamental invariants $(1,1)$. Then the orthogonal complement $V_2^\perp $ of $V_2$ in $\boldsymbol {V}$ admits a self-dual $O_E$-lattice $L_1$. Choose a normal basis (Definition 2.11) $\{e_1,\dots ,e_{2r-2}\}$ of $L_1$ under which the moment matrix is given by $\left (\begin {smallmatrix} 0 & u^{-1}\\ -u^{-1} & 0 \end {smallmatrix}\right )^{\oplus r-1}$. For every tuple $a=(a_1,\dots ,a_{2r-2})\in \mathbb {Z}^{2r-2}$ satisfying $a_{2i-1}+a_{2i}=0$ for $1\leqslant i\leqslant r-1$, the $O_E$-lattice

$$ \begin{align*} \Lambda_a := L_2\oplus\langle u^{a_1}e_1,\dots,u^{a_{2r-2}}e_{2r-2}\rangle \end{align*} $$

is integral with fundamental invariants $(0,\dots ,0,1,1)$ and, hence, vertex. It is clear that the intersection of all such $\Lambda _a$ is $L_2$. Since $r\geqslant 2$, the intersection of all $2$-dimensional nonsplit hermitian subspaces of $\boldsymbol {V}$ is $\{0\}$. Thus, the intersection of all vertex $O_E$-lattices of $\boldsymbol {V}$ is $\{0\}$.

Proof. Let $k'$ be the quadratic extension of k in $\overline {k}$. Note that $\mathcal {V}_\Lambda ^+$ has a canonical structure over $k'$, so that $\mathcal {V}_\Lambda ^{\circ +} := \mathcal {V}_\Lambda ^\circ \cap \mathcal {V}_\Lambda ^+$ (over $k'$) is the classical Deligne–Lusztig variety of $\operatorname {\mathrm {SO}}(\Lambda ^\vee /\Lambda )$ of Coxeter type.

Recall that $\delta $ is the Frobenius element of $\operatorname {\mathrm {Gal}}(\overline {k}/k)$. Fix a rational prime $\ell $ different from p. For every finite-dimensional $\overline {\mathbb {Q}}_\ell $-vector space V with an action by $\delta ^2$, we denote by $V^\dagger $ the subspace consisting of elements on which $\delta ^2$ acts by roots of unity. Then for the lemma, it suffices to show that for every $d\geqslant 0$, $\mathrm {H}_{2d}(\mathcal {V}_\Lambda ^+,\overline {\mathbb {Q}}_\ell (-d))^\dagger $ is generated by (the cycle class of) $\mathcal {V}_{\Lambda '}\cap \mathcal {V}_\Lambda ^+$ for all vertex $O_E$-lattices $\Lambda '$ containing $\Lambda $ with $t(\Lambda ')=2d+2$. By the same argument for [Reference Li and ZhangLZa, Theorem 5.3.2], it reduces to the following claim:

• (*) The action of $\delta ^2$ on $V := \bigoplus _{j\geqslant 0}\mathrm {H}^{2j}(\mathcal {V}_\Lambda ^{\circ +},\overline {\mathbb {Q}}_\ell (j))$ is semisimple and $V^\dagger =\mathrm {H}^0(\mathcal {V}_\Lambda ^{\circ +},\overline {\mathbb {Q}}_\ell )$.

There are three cases.

When $t(\Lambda )=2$, $\mathcal {V}_\Lambda ^{\circ +}$ is isomorphic to $\operatorname {\mathrm {Spec}}\overline {k}$; hence, ($*$) is trivial.

When $t(\Lambda )=4$, $\mathcal {V}_\Lambda ^{\circ +}$ is an affine curve; hence, ($*$) is again trivial.

When $t(\Lambda )\geqslant 6$, by Case (with $n=\tfrac {t(\Lambda )}{2}\geqslant 3$) in [Reference LusztigLus76, Section 7.3], the action of $\delta ^2$ on $\bigoplus _{j\geqslant 0}\mathrm {H}^j_c(\mathcal {V}_\Lambda ^{\circ +},\overline {\mathbb {Q}}_\ell )$ has eigenvalues $\{1,q^2,q^4,\dots ,q^{t(\Lambda )-2}\}$ and the eigenvalue $q^{2j}$ appears in $\mathrm {H}^{j+\frac {t(\Lambda )}{2}-1}_c(\mathcal {V}_\Lambda ^{\circ +},\overline {\mathbb {Q}}_\ell )$. Moreover by [Reference LusztigLus76, Theorem 6.1], the action of $\delta ^2$ is semisimple. Thus, ($*$) follows from the Poincaré duality.

The lemma is proved.

Proof. Let $s\in \mathcal {N}(\overline {k})$ be a closed point. By the wedge condition and the spin condition in Definition 2.1, we know that the map

$$ \begin{align*} \iota_X(u)-u\colon\operatorname{\mathrm{Lie}}(X)\otimes_{\mathscr{O}_{\mathcal{N}}}\mathscr{O}_{\mathcal{N},s}\to\operatorname{\mathrm{Lie}}(X)\otimes_{\mathscr{O}_{\mathcal{N}}}\mathscr{O}_{\mathcal{N},s} \end{align*} $$

has rank $1$ on both generic and special fibres. Thus, $F_X\otimes _{\mathscr {O}_{\mathcal {N}}}\mathscr {O}_{\mathcal {N},s}$ is a direct summand of $\operatorname {\mathrm {Lie}}(X)\otimes _{\mathscr {O}_{\mathcal {N}}}\mathscr {O}_{\mathcal {N},s}$ of rank $n-1$. The lemma follows.

Proof. This follows from the same proof for [Reference HowardHow19, Proposition 3.3].

Proof. This follows from the same proof for [Reference HowardHow19, Proposition 4.1].

Proof. The case $r=1$ has been proved in [Reference Rapoport, Smithling and ZhangRSZ17, Proposition 6.6]. Thus, we now assume $r\geqslant 2$.

We may assume that $\mathcal {N}(x)$ is nonempty. By the same argument in the proof of [Reference HowardHow19, Proposition 4.3], $\mathcal {N}(x)$ is locally defined by one equation. It remains to show that such an equation is not divisible by u. Since $r\geqslant 2$, this follows from [Reference KudlaKR11, Lemma 3.6], Lemma 2.4 and Corollary 2.31.

Proof of Proposition 2.33.

The proof of [Reference HowardHow19, Theorem 5.1] can be applied in the same way to Proposition 2.33, using Lemma 2.39 and Lemma 2.40.

Proof. Denote by $L^{\prime }_X$ the image of the map $\iota _X(u)-u\colon \operatorname {\mathrm {Lie}}(X)\to \operatorname {\mathrm {Lie}}(X)$. As we have $L^{\prime }_X\simeq \operatorname {\mathrm {Lie}}(X)/F_X$, $L^{\prime }_X$ is a locally free $\mathscr {O}_{\mathcal {N}}$-submodule of $\operatorname {\mathrm {Lie}}(X)$ of rank $1$ by Lemma 2.36. By the spin condition in Definition 2.1, for every closed point $s\in \mathcal {N}(\overline {k})$, the induced map $L^{\prime }_X\otimes _{\mathscr {O}_{\mathcal {N}}}\overline {k}\to \operatorname {\mathrm {Lie}}(X)\otimes _{\mathscr {O}_{\mathcal {N}}}\overline {k}$ over the residue field at s is injective. Thus, the quotient $\mathscr {O}_{\mathcal {N}}$-module $\operatorname {\mathrm {Lie}}(X)/L^{\prime }_X$ is locally free. It remains to show that $L^{\prime }_X\subseteq L_X$.

By definition, every section of $L^{\prime }_X$ can be locally written as the image of $(\iota _X(u)-u)x$ for some section x of $\mathrm {D}(X)$. We need to show that

1. (1) $\epsilon (\iota _X(u)-u)x$ is a section of $\operatorname {\mathrm {Fil}}(X)$;

2. (2) $(\epsilon (\iota _X(u)-u)x,y)=0$ for every section y of $F_X$.

For (1), we have $\epsilon (\iota _X(u)-u)x=(\iota _X(u)+u)(\iota _X(u)-u)x=(\iota _X(u^2)-u^2)x$. Since $\iota _X(u^2)-u^2$ acts by zero on $\operatorname {\mathrm {Lie}}(X)$, (1) follows.

For (2), we have $(\epsilon (\iota _X(u)-u)x,y)=((\iota _X(u)-u)x,(-\iota _X(u)+u)y)=0$ as y is a section of $\operatorname {\mathrm {ker}}(\iota _X(u)-u)$. Thus, (2) follows.

The lemma is proved.

Proof. Let $\delta $ be the Frobenius element of $\operatorname {\mathrm {Gal}}(\overline {k}/k)$.

Let $s\in \mathcal {N}(\overline {k})$ be a closed point represented by the quadruple $(X,\iota _X,\lambda _X;\rho _X)$. Let $\mathsf {M}$ be the covariant $O_F$-Dieudonné module of X equipped with the $O_E$-action $\iota _X$, which becomes a free $O_{\breve {E}}$-module. We have $\operatorname {\mathrm {Lie}}(X)=\mathsf {M}/\mathsf {V}\mathsf {M}$. By Definition 2.38 and Lemma 2.41, the fibre $\omega ^{-1}\mathbin {|}_s$ is canonically identified with $((u\otimes 1)\mathsf {M}+\mathsf {V}\mathsf {M})/\mathsf {V}\mathsf {M}$. By the identification between $\mathcal {V}_\Lambda $ and the generalised Deligne–Lusztig variety of $\operatorname {\mathrm {O}}(\Lambda ^\vee /\Lambda )$ in Proposition 2.28 given in [Reference WuWu, Proposition 4.29 & Proposition 5.13], we know that $\omega ^{-1}\mathbin {|}_{\mathcal {V}_\Lambda }$ coincides with $(\delta (U)+U)/U$ where U is the tautological subbundle of $(\Lambda ^\vee /\Lambda )\otimes _k\mathscr {O}_{\mathcal {V}_\Lambda }$.

To compute the degree of $(\delta (U)+U)/U$, let $\mathcal {V}_\Lambda ^+$ and $\mathcal {V}_\Lambda ^-$ be the two connected components of $\mathcal {V}_\Lambda $. Let $\mathcal {L}_\Lambda $ be the scheme over $\overline {k}$ classifying lines in $\Lambda ^\vee /\Lambda $ with the tautological bundle L. We may identify $\mathcal {V}_\Lambda ^+$ and $\mathcal {V}_\Lambda ^-$ as two closed subschemes of $\mathcal {L}_\Lambda $ via the assignment $U\mapsto \delta (U)\cap U$ (see [Reference Howard and PappasHP14, Section 3.2] for more details). Then, $\mathcal {V}_\Lambda ^+$ and $\mathcal {V}_\Lambda ^-$ are the two irreducible components of the locus where L and $\delta (L)$ generate an isotropic subspace and the assignment $L\mapsto \delta (L)$ switches $\mathcal {V}_\Lambda ^+$ and $\mathcal {V}_\Lambda ^-$. Let $\mathcal {I}_\Lambda $ be the locus where L is isotropic and $L=\delta (L)$. Then $\mathcal {I}_\Lambda $ is a disjoint union of $q^2+1$ copies of $\operatorname {\mathrm {Spec}}\overline {k}$ since there are exactly $q^2+1$ isotropic lines in $\Lambda ^\vee /\Lambda $ and is contained in $\mathcal {V}_\Lambda ^+\cap \mathcal {V}_\Lambda ^-$. Note that the map $\delta (U)/(\delta (U)\cap U)\to (\delta (U)+U)/U$ is an isomorphism and there is a short exact sequence