2.1 An elementary description

We give a down-to-earth description of groups $\tilde{G}$ , $G$ , the involution $\unicode[STIX]{x1D703}$ and the vertex $\mathbf{x}$ . Let $\tilde{V}$ be an $n$ -dimensional hermitian space over $E$ , spanned by basis vectors $\tilde{e}_{1},\ldots ,\tilde{e}_{n}$ and equipped with the hermitian form given by $\langle \sum a_{i}\tilde{e}_{i},\sum b_{i}\tilde{e}_{i}\rangle _{\text{herm}}=\sum _{i=1}^{n}a_{n+1-i}b_{i}^{\ast }$ , where $a_{i},b_{i}\in E$ and $b_{i}^{\ast }$ is the conjugate of $b_{i}$ over $F$ . Then $\tilde{G}$ is such an algebraic group defined over $F$ for which $\tilde{G}(F)$ is isomorphic to the group of unitary operators on $\tilde{V}$ , i.e.  $E$ -linear operators on $\tilde{V}$ preserving the hermitian form.

Let $\unicode[STIX]{x1D6EC}=\text{span}_{{\mathcal{O}}_{E}}\{\tilde{e}_{1},\ldots ,\tilde{e}_{n}\}$ be a lattice in $\tilde{V}$ . Let $K$ be the subgroup of $\tilde{G}(F)$ consisting of unitary operators $g$ with $g(\unicode[STIX]{x1D6EC})=\unicode[STIX]{x1D6EC}$ . Then $K$ stabilizes a unique vertex on the Bruhat–Tits building of $\tilde{G}$ over $F$ , which (up to conjugation) is the vertex that we call $\mathbf{x}$ . We have the stabilizer group $\tilde{G}(F)_{\mathbf{x}}=K$ .

The hermitian form $\langle \cdot \,,\cdot \rangle _{\text{herm}}$ takes ${\mathcal{O}}_{E}$ values on $\unicode[STIX]{x1D6EC}$ . Its reduction mod $\unicode[STIX]{x1D70B}^{1/2}$ thus defines a quadratic form $\langle \cdot \,,\cdot \rangle$ on $V:=\unicode[STIX]{x1D6EC}/\unicode[STIX]{x1D70B}^{1/2}\unicode[STIX]{x1D6EC}$ . Write $e_{1},\ldots ,e_{n}$ to be the reduction of $\tilde{e}_{1},\ldots ,\tilde{e}_{n}$ , respectively. Then $\langle \cdot \,,\cdot \rangle$ on $V$ is defined by $\langle \sum a_{i}e_{i},\sum b_{i}e_{i}\rangle =\sum _{i=1}^{n}a_{n+1-i}b_{i}$ , where $a_{i},b_{i}\in k$ . The algebraic group $G$ then should be identified with the group of automorphisms of $V$ (not necessarily fixing $\langle \cdot \,,\cdot \rangle$ ); $G(k^{\prime })=\text{GL}(V\otimes _{k}k^{\prime })$ for any finite extension $k^{\prime }/k$ , and $\unicode[STIX]{x1D703}\curvearrowright G(k^{\prime })$ is the involution $\unicode[STIX]{x1D703}(g)=(g^{\text{t}})^{-1}$ where $g^{\text{t}}$ denotes the transpose of $g$ with respect to the quadratic form $\langle \cdot \,,\cdot \rangle$ . The differential of $\unicode[STIX]{x1D703}$ , still denoted by $\unicode[STIX]{x1D703}$ acts on $\mathfrak{g}=\text{Lie }G$ by $\unicode[STIX]{x1D703}(X)=-X^{\text{t}}$ .

The Lie algebra $\tilde{\mathfrak{g}}(F)$ is the space of anti-hermitian endomorphisms of $\tilde{V}$ . For any $d\in \frac{1}{2}\mathbb{Z}$ , one has the attached Moy–Prasad sublattice $\tilde{\mathfrak{g}}(F)_{\mathbf{x},d}=\{X\in \tilde{\mathfrak{g}}(F)\mid X(\unicode[STIX]{x1D6EC})\subset \unicode[STIX]{x1D70B}^{d}\unicode[STIX]{x1D6EC}\}$ .

Note that the isomorphism above depends on the choice of uniformizer $\unicode[STIX]{x1D70B}^{1/2}\in E$ . Lastly, the algebraic group $G$ has $\unicode[STIX]{x1D703}$ -stable Borel subgroups defined over $k$ . One such $B$ is given by that $B(k)$ consists of endomorphisms of $V$ that sends $e_{i}$ to a linear combination of $e_{1}$ , $e_{2},\ldots ,$ and $e_{i}$ . We also denote $B(0)=B\cap G(0)=(B^{\unicode[STIX]{x1D703}})^{o}$ . They are used in § 4. For $n\geqslant 3$ , all such $\unicode[STIX]{x1D703}$ -stable Borel subgroups are $G(0)$ -conjugate.

3.1 Odd case

In this subsection we have $n=2g+1$ and $G=\text{GL}_{2g+1}/_{k}=\text{GL}(V)$ . Recall that the vector space $V$ comes with a non-degenerate quadratic form $\langle \cdot \,,\cdot \rangle$ . We then have in the introduction an involution $\unicode[STIX]{x1D703}$ on $G$ which sends $g$ to $(g^{\text{t}})^{-1}$ , where $g^{\text{t}}$ is the adjoint of $g$ with respect to $\langle \cdot \,,\cdot \rangle$ . This induces an involution on $\mathfrak{g}$ , and we write $\mathfrak{g}(0)=\mathfrak{g}^{\unicode[STIX]{x1D703}=1}$ , $\mathfrak{g}(1)=\mathfrak{g}^{\unicode[STIX]{x1D703}=-1}$ . We have $\mathfrak{g}(1)\cong \operatorname{Sym}^{2}(V)$ as $G(0)$ -representations. As $\langle \cdot \,,\cdot \rangle$ provides a self-dual structure on $V$ , $\mathfrak{g}(1)\cong \text{End}^{\text{self}-\text{adj}}(V)$ is also the space of self-adjoint operators on $V$ .

The representation $G(0)\curvearrowright \mathfrak{g}(1)$ , or equivalently $\text{SO}(V)\curvearrowright \operatorname{Sym}^{2}(V)$ , was considered by Bhargava and Gross in [Reference Bhargava and GrossBG14]. An orbit in this representation is GIT-stable if and only if it is contained in $\mathfrak{g}(1)^{\text{rs}}:=\mathfrak{g}^{\text{rs}}\cap \mathfrak{g}(1)$ where $\mathfrak{g}^{\text{rs}}$ is the open subset of regular semisimple elements in the Lie algebra. We now fix an $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ .

Let $p_{T}(x)$ be the degree $2g+1$ monic characteristic polynomial of $T$ . Let $L=k[x]/p_{T}(x)$ be a degree $2g+1$ étale algebra over $k$ . Consider the Weil restriction $\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ . This is a commutative étale finite group scheme over $k$ of order $2^{2g+1}$ . It has a surjective norm map $Nm:\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\rightarrow \unicode[STIX]{x1D707}_{2}$ . Bhargava and Gross observed for $T\in \mathfrak{g}(1)^{\text{rs}}$ , we have canonical isomorphism $\text{Stab}_{G(0)}(T)\cong \text{ker}(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . In fact, the map $T\mapsto p_{T}(x)$ is the GIT-quotient map $\mathfrak{g}(1)\mapsto \mathfrak{g}(1)/\!/G(0)$ ; we have $\mathfrak{g}(1)/\!/G(0)\cong \mathbb{A}^{2g+1}$ is the space of degree $n$ monic polynomials.

Let $C_{T}=(y^{2}=p_{T}(x))$ be a (smooth completion of) genus $g$ hyperelliptic curve. Let $J_{T}=\operatorname{Pic}^{0}(C_{T})$ . Since the $2$ -torsion $J_{T}[2]$ is generated by differences of Weierstrass points, one checks $J_{T}[2]\cong \text{ker}(\text{Res}_{k}^{k[x]/p_{T}(x)}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ . Consequently $J_{T}[2]\cong \text{Stab}_{G(0)}(T)$ .

If one fix such a $T$ , then the orbit of $T$ is $G(0)(k).T$ while the stable orbit of $T$ is $(G(0)(\bar{k}).T)\cap \mathfrak{g}(1)(\bar{k})$ . There could be more than one orbits inside a stable orbit, and relative to the choice of $T$ as a pinning they can be classified by $\ker (H^{1}(k,\text{Stab}_{G(0)}(T))\rightarrow H^{1}(k,G(0)))$ . When $k$ is a finite field, by Lang’s theorem, the latter pointed set is trivial, and thus we have $H^{1}(k,\text{Stab}_{G(0)}(T))\cong H^{1}(k,J_{T}[2])$ classifies orbits in the stable orbit of $T$ relative to the choice of a pinning.

The GIT-quotient map $\mathfrak{g}(1)\rightarrow \mathfrak{g}(1)/\!/G(0)$ has a Kostant section [Reference LevyLev09, Theorem 5.5]. Using the Kostant section as a pinning, a $G(0)(k)$ -orbit in $\mathfrak{g}(1)^{\text{rs}}(k)$ corresponds to a hyperelliptic curve $C_{T}$ together with a class in $H^{1}(k,J_{T}[2])$ . For $k$ a global field, Bhargava, Gross and others used this to study the average size of $2$ -Selmer groups of such hyperelliptic curves, see e.g. [Reference Bhargava and GrossBG13]. For this purpose, Wang developed the theory of pencil of quadrics [Reference WangWan13]. It turns out that his theory is very useful in describing the variety that we will encounter in orbital integrals.

On the vector space $V\oplus k$ we define two quadratic forms by $\langle (v_{1},c_{1}),(v_{2},c_{2})\rangle _{1}=\langle v_{1},v_{2}\rangle$ and $\langle (v_{1},c_{1}),(v_{2},c_{2})\rangle _{2}=\langle v_{1},Tv_{2}\rangle -c_{1}c_{2}$ . This defines a generic pencil of quadrics in the sense of Wang [Reference WangWan13, Introduction]. Recall that a subspace $W\subset V\oplus k$ is said to be isotropic with respect to a quadric (e.g.  $\langle \cdot \,,\cdot \rangle _{1}$ ) if the restriction of the quadratic form to $W$ is trivial. In his paper, Wang proved the following result.

Theorem 3.1 (Wang [Reference WangWan13, Theorem 2.26]).

Let $F_{T}$ be the variety that parameterizes $g$ -dimensional subspaces of $V\oplus k$ that the are isotropic with respect to both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ . Then there is a commutative algebraic group structure on

$$\begin{eqnarray}G_{T}:=J_{T}\sqcup F_{T}\sqcup \operatorname{Pic}^{1}(C_{T})\sqcup F_{T}^{\prime },\end{eqnarray}$$

where $F_{T}^{\prime }$ is a copy of $F_{T}$ as an abstract variety, and $G_{T}$ satisfies the following.

1. (i) The component group of $G_{T}$ is equal to $\mathbb{Z}/4$ . The four components above correspond to $0+4\mathbb{Z}$ , $1+4\mathbb{Z}$ , $2+4\mathbb{Z}$ and $3+4\mathbb{Z}$ , respectively.

2. (ii) The addition law on $J_{T}\sqcup \operatorname{Pic}^{1}(C_{T})$ agrees with the natural one on $\operatorname{Pic}(C_{T})/(2(\infty )=0)$ .

3. (iii) The inversion map of $G_{T}$ restricts to an isomorphism $F_{T}\xrightarrow[]{{\sim}}F_{T}^{\prime }$ .

In particular, $F_{T}$ is a torsor under $J_{T}$ and there is a doubling map $\times 2:F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ . We review the group structure in the theorem. The group structure is determined by $(p)-[W]$ , i.e. how to subtract from $p\in C_{T}$ a subspace $[W]\in F_{T}$ . This is done as follows: recall that a ruling is a connected component of the variety parameterizing $(g+1)$ -dimensional subspace on which the quadratic form is trivial. A point $p=(x,y)$ on $C_{T}$ corresponds to a ruling of $\langle \cdot \,,\cdot \rangle _{2}-x\langle \cdot \,,\cdot \rangle _{1}$ (see [Reference WangWan13, p. 8]). There will be a unique $(g+1)$ -dimensional space $W^{\prime }$ in the ruling such that $W^{\prime }\supset W$ . Inside the space $W^{\prime }$ there will be, when counted with multiplicity, two $g$ -dimensional subspaces $W$ and $W^{\prime \prime }$ on which both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish. It is then defined $(p)-[W]:=[W^{\prime \prime }]$ . This uniquely characterizes the group structure on $G_{T}$ .

We give an example of $(p)-[W]$ when $p=\infty$ , the rational point on $C_{T}$ at infinity. Note that $\infty \in C_{T}$ correspond to the quadric $\langle \cdot \,,\cdot \rangle _{1}$ , which is degenerate and has only one ruling. Let $\unicode[STIX]{x1D70F}_{\infty }:V\oplus k\rightarrow V\oplus k$ be the map that sends $(v,c)$ to $(v,-c)$ . It is obvious that if $[W]\in F_{T}$ , then $[\unicode[STIX]{x1D70F}_{\infty }W]\in F_{T}$ . Let us fix such $W$ and write $W^{\prime \prime }=\unicode[STIX]{x1D70F}_{\infty }W$ . As $W\subset V\oplus k$ is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{2}$ , the second component is not contained in $W$ . Hence, $W^{\prime }:=W+k$ , where this $k$ is the second component of $V\oplus k$ , is $(g+1)$ -dimensional, and is isotropic with respect to $\langle \cdot \,,\cdot \rangle _{1}$ as $W$ is. This says that $W^{\prime }$ and $W^{\prime \prime }$ satisfy the properties in the previous paragraph, and consequently $(\infty )-[W]=[W^{\prime \prime }]=[\unicode[STIX]{x1D70F}_{\infty }(W)]$ .

It is obvious that $G_{T}$ depends only on $T$ up to $G(0)(k)$ -conjugacy. As mentioned in the introduction the orbit of $T$ in its stable orbit may be characterized by a class $\unicode[STIX]{x1D6FC}_{T}\in H^{1}(k,J_{T}[2])$ . This class can be described as follows: by Theorem 3.1(i) and (ii) the map $\times 2:F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ is étale and Galois with Galois group being $J_{T}[2]$ as a group scheme over $k$ . From the previous paragraph and [Reference WangWan13, Proposition 2.29], respectively, one has the following result.

We will later need an explicit description of the $J_{T}[2]$ -action on $(\times 2)^{-1}(\infty )$ . Let $p_{0},\ldots ,p_{2g},\infty$ be the Weierstrass points of $C_{T}$ . Then $J_{T}[2]$ is generated by $((p_{i})-(\infty ))$ , $0\leqslant i\leqslant 2g$ with the only relation $\sum _{i=0}^{2g}((p_{i})-(\infty ))=0$ . The action of $(p_{i})-(\infty )$ can be described as follows. Say $p_{i}=(x,0)$ . Then $x\langle \cdot \,,\cdot \rangle _{1}-\langle \cdot \,,\cdot \rangle _{2}$ is a degenerate quadric with one-dimensional kernel $U$ . For $[W]\in (\times 2)^{-1}(\infty )$ , $x\langle \cdot \,,\cdot \rangle _{1}-\langle \cdot \,,\cdot \rangle _{2}$ is trivial on the $(g+1)$ -dimensional space $W+U$ . There is exactly one $g$ -dimensional subspace $W^{\prime }\subset W+U$ , other than $W$ , on which $\langle \cdot \,,\cdot \rangle$ and $\langle \cdot ,T\cdot \rangle$ are also trivial. The $J_{T}[2]$ -action on $(\times 2)^{-1}(\infty )$ is then given by the following result.

For any $0\leqslant m\leqslant g$ , consider $j_{m}:\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by $j_{m}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-(m-1)(\infty )$ . Let $X_{T,m}$ be the image of $j_{m}$ , and let $\tilde{X}_{T,m}:=(\times 2)^{-1}(X_{T,m})$ be its preimage under the étale map $\times 2$ . For example $\tilde{X}_{T,0}=(\times 2)^{-1}(\infty )$ . We also put $\tilde{X}_{T,-1}=\emptyset$ . We shall relate $\tilde{X}_{T,m}$ with the following varieties $F_{T,m}$ , which could be thought as a generalized version of Hessenberg varieties considered by Goresky et al. [Reference Goresky, Kottwitz and MacPhersonGKM06].

For any finite extension $k^{\prime }/k$ , we call a flag of $k^{\prime }$ -subspaces $0=W^{0}\subset W^{1}\subset \cdots \subset W^{g}\subset (V\oplus k)\otimes _{k}k^{\prime }$ good if:

1. (i) $\dim W^{i}=i$ ;

2. (ii) the restriction of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ to $W^{g}$ is zero;

3. (iii) $W^{g-1}\subset V\otimes _{k}k^{\prime }$ ;

4. (iv) $T(W^{i})\subset W^{i+2}$ for all $1\leqslant i\leqslant g-3$ ;

5. (v) $T(W^{g-2})\subset \unicode[STIX]{x1D70B}_{1}(W^{g})$ .

Here $\unicode[STIX]{x1D70B}_{1}:(V\oplus k)\otimes _{k}k^{\prime }\rightarrow V\otimes _{k}k^{\prime }$ is the projection to the first factor. For $1\leqslant m\leqslant g$ , a good flag is called $m$ -good if $T(W^{g-m})\subset \unicode[STIX]{x1D70B}_{1}(W^{g-m+1})$ . Note this is simply $T(W^{g-m})\subset W^{g-m+1}$ when $m>1$ . Also a good flag is called $0$ -good if $W^{g}\subset V\otimes _{k}k^{\prime }$ . Next, for $0\leqslant m\leqslant g$ , an $m$ -good flag is called $m$ -excellent if it is also $n$ -good for $m<n\leqslant g$ . On the other hand, a good flag is called $m$ -general if it is not $n$ -good for any $0\leqslant n<m$ . Finally, a good flag is called $m$ -exact if it is $m$ -excellent and $m$ -general. Now let

$$\begin{eqnarray}F_{T,m}(k^{\prime })=\{0=W^{0}\subset W^{1}\subset \cdots \subset W^{g}\subset (V\oplus k)\otimes _{k}k^{\prime }\mid \text{This is an }m\text{-exact flag}\}.\end{eqnarray}$$

The functor $F_{T,m}$ is easily seen from its very definition to be represented by a quasi-projective variety over $k$ which we will denote with the same notation. In fact, there is a projective variety $F_{T,\text{good}}$ that parameterize good flags, and $F_{T,m}\subset F_{T,\text{good}}$ is locally closed. There is a natural map $\tilde{j}:F_{T,\text{good}}\rightarrow F_{T}$ by sending a flag to $[W^{g}]$ . In the rest of this section, we work ‘geometrically,’ i.e. we replace $k$ by an algebraic closure $\bar{k}$ , so that we can omit the notation $\cdot \otimes _{k}k^{\prime }$ and so on. This section is mostly devoted to the proof of the following result.

We now begin our proof for the general case with two simple lemmas.

Let $F_{T,m}^{\text{ex}}\subset F_{T,\text{good}}$ be projective varieties parameterizing $m$ -excellent flags. The key is as follows.

Proof of Lemma 3.8.

Recall $\tilde{X}_{T,0}=(\times 2)^{-1}(\infty )$ by definition. By Lemma 3.2(i), $(\times 2)^{-1}(\infty )$ parameterizes $g$ -dimensional varieties $W^{g}$ in $V$ on which $\langle \cdot \,,\cdot \rangle _{1}|_{V}=\langle \cdot \,,\cdot \rangle$ and $\langle \cdot \,,\cdot \rangle _{2}|_{V}=\langle \cdot ,T\cdot \rangle$ vanish. Lemma 3.7(ii) and (iii) then says it extends uniquely to a $0$ -exact flag (there we began with a flag rather than $W^{g}$ itself, but the same proof applies). This proves the $m=0$ statement.

From now on $0<m\leqslant g$ is fixed. We shall show $\tilde{j}(F_{T,m}^{\text{ex}})\subset \tilde{X}_{T,m}$ for a generic $T\in \mathfrak{g}(1)^{\text{rs}}$ (i.e. for $T$ in a Zariski open subset of $\mathfrak{g}(1)^{\text{rs}}$ ). In fact, what we will do is the following: fix a flag $\mathbb{F}=(0\subset W^{1}\subset \cdots \subset W^{g-1}\subset W^{g}\subset V\oplus k)$ such that $W^{g-1}\subset V$ , $W^{g}\not \subset V$ , $k\not \subset W^{g}$ (here $V$ and $k$ are the first and the second components in $V\oplus k$ ), and $\langle \cdot \,,\cdot \rangle$ is trivial on $\unicode[STIX]{x1D70B}_{1}(W^{g})$ . Since all such flags in $V\oplus k$ are conjugate by $G(0)$ (where $G(0)$ preserves $V\subset V\oplus k$ and acts trivially on the second component), without loss of generality we may assume that $\mathbb{F}$ is exactly the flag in interest.

There is an irreducible closed subvariety ${\mathcal{V}}\subset \mathfrak{g}(1)$ such that $\mathbb{F}$ is $m$ -excellent with respect to $T\in \mathfrak{g}(1)^{\text{rs}}$ if and only if $T$ lies inside ${\mathcal{V}}$ . There is an Zariski open subset of ${\mathcal{V}}$ consisting of those $T$ for which the flag is $m$ -exact. What we shall prove is that for an even smaller open subset $\unicode[STIX]{x1D6E5}\subset {\mathcal{V}}$ , all $T\in \unicode[STIX]{x1D6E5}$ satisfy $\tilde{j}(\mathbb{F})\in \tilde{X}_{T,m}$ . A continuity argument by having Theorem 3.1 in family then extends the result to all $T\in {\mathcal{V}}$ , which is what we need.

The case $m=g$ is trivial, and we will assume $0<m<g$ . Let $0\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ be an $m$ -good flag. Consider $U_{0}=W^{g-m}$ and $U^{0}=(W^{g-m+1})^{\bot 1}:=\{(v,c)\in V\oplus k\mid \langle v,w\rangle =0$ , for all $w\in W^{g-m+1}\}$ (that is, $^{\bot 1}$ is used to denote the orthogonal complement with respect to $\langle \cdot \,,\cdot \rangle _{1}$ ). Following the spirit of [Reference WangWan13, § 3.1], we define inductively subspaces $U_{0}\subset U_{1}\subset \cdots \subset U_{\lfloor (m+1)/2\rfloor }\subset U^{\lfloor (m+1)/2\rfloor }\subset U^{\lfloor (m+1)/2\rfloor -1}\subset \cdots \subset U^{0}$ as follows:

$$\begin{eqnarray}\left\{\begin{array}{@{}l@{}}U_{n}:=(U^{n-1})^{\bot 1}\cap W^{g-m+2n-1},\\ U^{n}:=(U_{n})^{\bot 2}\cap U^{n-1}.\end{array}\right.\end{eqnarray}$$

We now come back to the proof of Lemma 3.8. Define $L$ to be the variety that parameterize $g$ -dimensional subspaces $W$ satisfying $U_{\lfloor (m+1)/2\rfloor }\subset W\subset U^{\lfloor (m+1)/2\rfloor }$ and that $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ vanish on $W$ . In particular, by construction we have $U_{\lfloor (m+1)/2\rfloor }\subset W^{g-m+2\lfloor (m+1)/2\rfloor -1}\subset W^{g}$ and $W^{g}\subset (W^{g})^{\bot 2}\cap U^{0}\subset U^{\lfloor (m+1)/2\rfloor }$ , i.e.  $[W^{g}]\in L$ .

Define $\overline{V}$ to be the subquotient $\overline{V}\,:=\,U^{\lfloor (m+1)/2\rfloor }/U_{\lfloor (m+1)/2\rfloor }$ . Since $U_{\lfloor (m+1)/2\rfloor }\subset (U^{\lfloor (m+1)/2\rfloor -1})^{\bot 1}\subset (U^{\lfloor (m+1)/2\rfloor })^{\bot 1}$ and $U^{\lfloor (m+1)/2\rfloor }\subset (U_{\lfloor (m+1)/2\rfloor })^{\bot 2}\Rightarrow U_{\lfloor (m+1)/2\rfloor }\subset (U^{\lfloor (m+1)/2\rfloor })^{\bot 2}$ , the two quadratic forms $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ restricts to be quadratic forms on  $\overline{V}$ . Denote still their restrictions by $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ , respectively. Let $L^{\prime }$ be the variety that parameterize $\lfloor m/2\rfloor$ -dimensional subspaces in $\overline{V}$ on which $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ are trivial. Then we have $\unicode[STIX]{x1D704}_{T}:L\xrightarrow[]{{\sim}}L^{\prime }$ which sends a $g$ -dimensional subspace of $V$ contained in $U^{\lfloor (m+1)/2\rfloor }$ to its image in $\overline{V}$ .

Consider the polynomials $p_{T}^{(i)}(x)=\text{disc}(\langle \cdot \,,\cdot \rangle _{1}-x\langle \cdot \,,\cdot \rangle _{2})|_{U^{i}/U_{i}}$ for $i=0,1,\ldots ,\lfloor (m+1)/2\rfloor$ . We claim $p_{T}^{(0)}(x)=x^{2\lfloor (m+1)/2\rfloor }p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ . To see this, observe that when we go from $U^{0}/U_{0}$ to $U^{1}/U_{1}$ , we quotient out $U_{1}/U_{0}$ , on which both $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ is zero. Even more, $U_{1}/U_{0}$ is in the kernel of $\langle \cdot \,,\cdot \rangle _{1}$ while $\langle U_{1}/U_{0},U^{0}/U^{1}\rangle _{2}$ is non-trivial. This exactly says $p_{T}^{(0)}(x)=x^{2}p_{T}^{(1)}(x)$ . Repeating the argument gives the asserted result. We also note $\deg p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)=2m-2\lfloor (m+1)/2\rfloor +1=2\lfloor m/2\rfloor +1$ .

Let $\bar{C}_{T}$ be the hyperelliptic curve given by $(y^{2}=p_{T}^{(\lfloor (m+1)/2\rfloor )}(x))$ with completion smooth at infinity. Recall $T\in {\mathcal{V}}$ is such that our flag $\mathbb{F}$ is $m$ -excellent with respect to $T$ . One checks from definition that when $T$ runs over the subspace ${\mathcal{V}}$ , $p_{T}^{(0)}(x)$ runs over all monic polynomials of degree less than or equal to $2m+1$ that are divided by $x^{m+1}$ . Consequently, there exists a Zariski open subset $\unicode[STIX]{x1D6E5}\subset {\mathcal{V}}$ such that for $T\in \unicode[STIX]{x1D6E5}$ , $p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is separable (i.e. having distinct roots) and $\bar{C}_{T}$ is smooth.

For such $T$ , the relation between $L^{\prime }$ and $\bar{C}_{T}$ is exactly that between $(\times 2)^{-1}(\infty )$ and $C_{T}$ in Lemma 3.2(i). In particular, we have a simply transitive $\operatorname{Pic}^{0}(\bar{C}_{T})[2]$ -action on $L^{\prime }$ described by Lemma 3.3. Let $\overline{\unicode[STIX]{x1D6FA}}_{0}\subset \overline{V}$ be the $\lfloor m/2\rfloor$ -dimensional subspace that corresponds to $W^{g}$ , i.e.  $\unicode[STIX]{x1D704}_{T}([W^{g}])=[\overline{\unicode[STIX]{x1D6FA}}_{0}]$ . Let $\bar{p}_{0},\ldots ,\bar{p}_{2\lfloor m/2\rfloor },\bar{\infty }$ be the Weierstrass points of $\bar{C}_{T}$ , the last one understood as the point at infinity. We define $\overline{\unicode[STIX]{x1D6FA}}_{i}$ for $i=1,2,\ldots ,2\lfloor m/2\rfloor +1$ by $[\overline{\unicode[STIX]{x1D6FA}}_{i}]=((\bar{p}_{i-1})-(\bar{\infty })).[\overline{\unicode[STIX]{x1D6FA}}_{i-1}]$ . By Lemma 3.3, $\overline{\unicode[STIX]{x1D6FA}}_{i-1}$ and $\overline{\unicode[STIX]{x1D6FA}}_{i}$ intersect in codimension one. We note that the relation $\sum ((\bar{p}_{i})-(\bar{\infty }))=0$ implies $\overline{\unicode[STIX]{x1D6FA}}_{2\lfloor m/2\rfloor +1}=\overline{\unicode[STIX]{x1D6FA}}_{0}$ .

Now we pass this sequence from $L^{\prime }$ to $L$ . Let $\unicode[STIX]{x1D6FA}_{i}\subset V\oplus k$ be the preimage of $\overline{\unicode[STIX]{x1D6FA}}_{i}$ for each $i$ . We also have $\unicode[STIX]{x1D6FA}_{i-1}$ and $\unicode[STIX]{x1D6FA}_{i}$ intersect in codimension one for each $i=1,2,\ldots ,2\lfloor m/2\rfloor +1$ . This will allow us to fetch the precise information we want about $[\unicode[STIX]{x1D6FA}_{i}]$ using the description of the group structure of $G_{T}$ following Theorem 3.1: the subspace $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ is $(g+1)$ -dimensional, and any quadratic form among $x_{1}\langle \cdot \,,\cdot \rangle _{1}+x_{2}\langle \cdot \,,\cdot \rangle _{2}$ factor through a linear pairing of $(\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i})/\unicode[STIX]{x1D6FA}_{i-1}$ and $(\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i})/\unicode[STIX]{x1D6FA}_{i}$ since the quadratic form is trivial on both $\unicode[STIX]{x1D6FA}_{i-1}$ and $\unicode[STIX]{x1D6FA}_{i}$ by assumption. Consequently some non-trivial combination of $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ must vanish on $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ . The space $\unicode[STIX]{x1D6FA}_{i-1}$ , $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ and $\unicode[STIX]{x1D6FA}_{i}$ then play the role of $W$ , $W^{\prime }$ and $W^{\prime \prime }$ in the paragraph following Theorem 3.1, respectively, which means there exists some $p_{i}\in C_{T}$ such that $(p_{i})-[\unicode[STIX]{x1D6FA}_{i-1}]=[\unicode[STIX]{x1D6FA}_{i}]$ .

Recall that $\unicode[STIX]{x1D6FA}_{2\lfloor m/2\rfloor +1}=\unicode[STIX]{x1D6FA}_{0}=W^{g}$ . Hence, we have

(3.1) $$\begin{eqnarray}((p_{1})-((p_{2})-\cdots -((p_{2\lfloor m/2\rfloor +1})-[W^{g}])\cdots \,))=[W^{g}].\end{eqnarray}$$

Put $p_{i}^{\ast }$ to be the hyperelliptic conjugate of $p_{i}$ for $i=2,4,\ldots ,2\lfloor m/2\rfloor$ and $p_{i}^{\ast }=p_{i}$ for odd $i$ , so that $(p_{i}^{\ast })-(\infty )=(-1)^{i+1}((p_{i})-(\infty ))\in J_{T}$ . Equation (3.1) then becomes

$$\begin{eqnarray}\mathop{\sum }_{i=1}^{2\lfloor m/2\rfloor +1}(p_{i}^{\ast })-2\biggl\lfloor\frac{m}{2}\biggr\rfloor(\infty )-[W^{g}]=[W^{g}]\in F_{T}\subset G_{T}\end{eqnarray}$$

or, equivalently,

(3.2) $$\begin{eqnarray}2[W^{g}]=\mathop{\sum }_{i=1}^{2\lfloor m/2\rfloor +1}(p_{i}^{\ast })-2\biggl\lfloor\frac{m}{2}\biggr\rfloor(\infty )\in \operatorname{Pic}^{1}(C_{T})\subset G_{T}\end{eqnarray}$$

which says $(\times 2)([W^{g}])\in X_{T,2\lfloor m/2\rfloor +1}$ . This concludes the proof of Lemma 3.8 when $m$ is odd.

When $m$ is even, the fact that $p_{T}^{(0)}(x)=x^{2\lfloor (m+1)/2\rfloor }p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is divisible by $x^{m+1}$ implies that $p_{T}^{(\lfloor (m+1)/2\rfloor )}(x)$ is divisible by $x$ . This means $(0,0)=\bar{p}_{i}$ is one of the Weierstrass point of  $\bar{C}_{T}$ . Recall $[\overline{\unicode[STIX]{x1D6FA}}_{i}]=((\bar{p}_{i})-(\infty )).[\overline{\unicode[STIX]{x1D6FA}}_{i-1}]$ . By Lemma 3.3 we know $\langle \cdot \,,\cdot \rangle _{1}$ vanish on $\overline{\unicode[STIX]{x1D6FA}}_{i-1}+\overline{\unicode[STIX]{x1D6FA}}_{i}$ , and thus $\langle \cdot \,,\cdot \rangle _{1}$ also vanish on the preimage $\unicode[STIX]{x1D6FA}_{i-1}+\unicode[STIX]{x1D6FA}_{i}$ . But this says $p_{i}=\infty$ . Thus, (3.2) gives the better result $(\times 2)([W^{g}])\in X_{T,2\lfloor m/2\rfloor }=X_{T,m}$ and we are done.◻

3.2 Even case

In this subsection $n=2g+2$ ; $\tilde{G}=U_{2g+2}(E/F)$ , $G=\text{GL}_{2g+2}/_{k}=\text{GL}(V)$ , $G(0)=\text{SO}(V)$ and $\mathfrak{g}(1)=\operatorname{Sym}^{2}(V)$ where $V$ is a $(2g+2)$ -dimensional non-degenerate split quadratic space. The method in this subsection is almost identical to that of the previous one, and we only list the setting, definition and results here.

We have parallel result to Theorem 3.4. Again fix $T\in \mathfrak{g}(1)^{\text{rs}}(k)$ . We also write $p_{T}(x)\in k[x]$ the monic characteristic polynomial of $T$ . Our hyperelliptic curve $C_{T}:=(y^{2}=p_{T}(x))$ now has two points above the infinity on $\mathbb{P}^{1}$ . We shall denote these two points by $\infty ^{(1)}$ and $\infty ^{(2)}$ . They are both defined over $k$ .

Consider $L=k[x]/p_{T}(x)$ . The Weil restriction $\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ now has not only a surjective norm map $Nm\,:\,\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\rightarrow \unicode[STIX]{x1D707}_{2}$ but also a diagonal embedding $\unicode[STIX]{x1D6E5}\,:\,\unicode[STIX]{x1D707}_{2}\rightarrow \text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ . We have $\text{Stab}_{\text{O}(V)}(T)\cong \text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}$ , also $\text{Stab}_{G(0)}(T)\cong \ker (\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2})$ , and lastly $J_{T}[2]\cong \text{Stab}_{G(0)}(T)/Z(G(0))\cong (\ker (\text{Res}_{k}^{L}\unicode[STIX]{x1D707}_{2}\xrightarrow[]{Nm}\unicode[STIX]{x1D707}_{2}))/\unicode[STIX]{x1D6E5}(\unicode[STIX]{x1D707}_{2})$ .

We denote by $\langle \cdot \,,\cdot \rangle _{1}=\langle \cdot \,,\cdot \rangle$ the standard quadratic form on $V$ , i.e. the one which is invariant by $G(0)$ . Then $\infty ^{(1)}$ and $\infty ^{(2)}$ are just the two rulings of $\langle \cdot \,,\cdot \rangle _{1}$ . Define $\langle \cdot \,,\cdot \rangle _{2}$ on $V$ by $\langle v_{1},v_{2}\rangle _{2}=\langle v_{1},Tv_{2}\rangle _{1}$ . Then the theory of pencil of quadrics says the following.

Theorem 3.12 (Wang [Reference WangWan13, Theorem 2.26]).

Let $F_{T}$ be the variety that parameterizes $g$ -dimensional subspaces of $V$ that are isotropic with respect to $\langle \cdot \,,\cdot \rangle _{1}$ and $\langle \cdot \,,\cdot \rangle _{2}$ . Then there is a commutative algebraic group structure on

$$\begin{eqnarray}G_{T}:=J_{T}\sqcup F_{T}\sqcup \operatorname{Pic}^{1}(C_{T})\sqcup F_{T}^{\prime },\end{eqnarray}$$

where $F_{T}^{\prime }$ is a copy of $F_{T}$ as an abstract variety, and $G_{T}$ satisfies:

1. (i) $G_{T}$ has component group equal to $\mathbb{Z}/4$ ; the four components above correspond to $0+4\mathbb{Z}$ , $1+4\mathbb{Z}$ , $2+4\mathbb{Z}$ and $3+4\mathbb{Z}$ , respectively;

2. (ii) the addition law on $J_{T}\sqcup \operatorname{Pic}^{1}(C_{T})$ agrees with the natural one on $\operatorname{Pic}(C_{T})/((\infty ^{(1)})+(\infty ^{(2)})=0\!)$ ;

3. (iii) the inversion map of $G_{T}$ restricts to an isomorphism $F_{T}\xrightarrow[]{{\sim}}F_{T}^{\prime }$ .

We again write the doubling map $\times 2\,:\,F_{T}\rightarrow \operatorname{Pic}^{1}(C_{T})$ which is étale Galois with Galois group $J_{T}[2]$ . For $0\leqslant m\leqslant g$ with $m$ even, define $j_{m}^{(1)},j_{m}^{(2)}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by

$$\begin{eqnarray}j_{m}^{(1)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\bigg(\frac{m}{2}-1\bigg)(\infty ^{(1)})-\frac{m}{2}(\infty ^{(2)}),\end{eqnarray}$$

$$\begin{eqnarray}j_{m}^{(2)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m}{2}(\infty ^{(1)})-\bigg(\frac{m}{2}-1\bigg)(\infty ^{(2)}),\end{eqnarray}$$

and we define $X_{T,m}^{(i)}$ to be the image of $j_{m}^{(i)}$ , and $\tilde{X}_{T,m}^{(i)}=(\times 2)^{-1}(X_{T,m}^{(i)})$ , $i=1,2$ .

For $0<m\leqslant g$ with $m$ odd, we define $j_{m}^{(0)},j_{m}^{(1)},j_{m}^{(2)}\,:\,\operatorname{Sym}^{m}(C_{T})\rightarrow \operatorname{Pic}^{1}(C_{T})$ by

$$\begin{eqnarray}j_{m}^{(0)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m-1}{2}(\infty ^{(1)})-\frac{m-1}{2}(\infty ^{(2)}),\end{eqnarray}$$

$$\begin{eqnarray}j_{m}^{(1)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m-3}{2}(\infty ^{(1)})-\frac{m+1}{2}(\infty ^{(2)}),\end{eqnarray}$$

$$\begin{eqnarray}j_{m}^{(2)}(p_{1},\ldots ,p_{m})=(p_{1})+\cdots +(p_{m})-\frac{m+1}{2}(\infty ^{(1)})-\frac{m-3}{2}(\infty ^{(2)}),\end{eqnarray}$$

and we define $X_{T,m}^{(i)}$ to be the image of $j_{m}^{(i)}$ , and $\tilde{X}_{T,m}^{(i)}=(\times 2)^{-1}(X_{T,m}^{(i)})$ , $i=0,1,2$ .

Next, we introduce the notion of good flags. A flag of subspaces $0\subset W^{1}\subset \cdots \subset W^{g+1}\subset V$ is called good if:

1. (i) $\dim W^{i}=i$ ;

2. (ii) the restriction of $\langle \cdot \,,\cdot \rangle _{1}$ to $W^{g+1}$ is zero;

3. (iii) the restriction of $\langle \cdot \,,\cdot \rangle _{2}$ to $W^{g}$ is zero;

4. (iv) $T(W^{i})\subset W^{i+2}$ , for all $1\leqslant i\leqslant g-1$ .

For $0\leqslant m\leqslant g$ , a good flag is called $m$ -good if $T(W^{g-m})\subset W^{g-m+1}$ . Here $W^{-1}=W^{0}=\{0\}$ , i.e. good flags are automatically $g$ -good. A flag is called $m$ -excellent if it is $n$ -good for $m\leqslant n\leqslant g$ .

For $0\leqslant m\leqslant g$ , a good flag is called $m$ -general if it is not $n$ -good for any $0\leqslant n<m$ . For any $0<m\leqslant g$ , we now define the notion of $m$ -exact flags (see also Remark 3.14). Let $\{W^{r}\}_{r=1}^{g+1}$ be any $m$ -excellent and $m$ -general flag. There always exists another $m$ -excellent flag $\{U^{r}\}_{r=1}^{g+1}$ satisfying $U^{g}=W^{g}$ but $U^{g+1}\not =W^{g+1}$ if $m$ is odd, or $U^{g-1}=W^{g-1}$ , $U^{g+1}=W^{g+1}$ but $U^{g}\not =W^{g}$ if $m$ is even. We say $\{W^{r}\}_{r=1}^{g+1}$ is $m$ -exact if $\{U^{r}\}_{r=1}^{g+1}$ is also $m$ -general. Lastly, a $0$ -excellent flag is said to be $0$ -exact.

Now let $F_{T,m}^{(1)}$ be the variety that parameterize $m$ -exact flags for which $W^{g+1}$ is in the ruling $\infty ^{(1)}$ , and $F_{T,m}^{(2)}$ be the variety that parameterizes those $m$ -exact flags for which $W^{g+1}$ is in the other ruling $\infty ^{(2)}$ . We have natural maps $\tilde{j}\,:\,F_{T,m}\,:=\,F_{T,m}^{(1)}\sqcup F_{T,m}^{(2)}\rightarrow F_{T}$ by sending $\{W^{r}\}_{r=1}^{g+1}$ to $W^{g}$ .

Theorem 3.13. For $0\leqslant m\leqslant g$ , the restriction of $\tilde{j}$ to $F_{T,m}^{(1)}$ is a locally closed embedding, with image equal to

$$\begin{eqnarray}\begin{array}{@{}cc@{}}\tilde{X}_{T,m}^{(1)}\backslash (\tilde{X}_{T,m-1}^{(0)}\cup \tilde{X}_{T,m-1}^{(1)}) & \text{if }m\text{ is even},\\ \tilde{X}_{T,m}^{(0)}\backslash (\tilde{X}_{T,m-1}^{(1)}\cup \tilde{X}_{T,m-1}^{(2)}) & \text{if }m\text{ is odd},\\ \end{array}\end{eqnarray}$$

where for $F_{T,m}^{(2)}$ , we replace, in the case $m$ is even, the two superscripts $^{(1)}$ by $^{(2)}$ .

4.1 Geometric identification

The goal in this subsection is to prove Proposition 4.3. We begin with a lemma.

Proof. For this proof only we will replace $\tilde{G}$ by $\text{SU}_{2g+1}(E/F)$ . One checks that this replacement does not affect the orbits. Now the orbits in the stable orbit of $T$ are classified by $\ker (H^{1}(k,\text{Stab}_{G(0)}(T))\rightarrow H^{1}(k,G(0)))$ and that of $\tilde{T}$ by $\ker (H^{1}(F,\text{Stab}_{\tilde{G}}(\tilde{T}))\rightarrow H^{1}(F,\tilde{G}))$ . By Lang’s theorem and the fact that simply connected group over a non-archimedean local field has trivial $H^{1}$ , we have $H^{1}(k,G(0))=H^{1}(F,\tilde{G})=0$ . Recall also that $\text{Stab}_{G(0)}(T)\cong J_{T}[2]$ .

The key is that our $\tilde{T}$ has its centralizer $\tilde{G}_{\tilde{T}}$ is anisotropic over $F^{ur}$ , the maximal unramified extension of $F$ [Reference TsaiTsa15a, Theorem 2.1]. Consider the exact sequence

$$\begin{eqnarray}1\rightarrow H^{1}(\text{Gal}(F^{ur}/F),\text{Stab}_{\tilde{G}}(\tilde{T})(F^{ur}))\rightarrow H^{1}(F,\text{Stab}_{\tilde{G}}(\tilde{T}))\rightarrow H^{1}(F^{ur},\text{Stab}_{\tilde{G}}(\tilde{T})).\end{eqnarray}$$

The last cohomology group is trivial by Steinberg’s theorem. The first cohomology group is isomorphic to $H^{1}(k,J_{T}[2])$ because $J_{T}[2](\bar{k})$ is a quotient of $\text{Stab}_{\tilde{G}}(\tilde{T})(F^{ur})$ with kernel possessing a filtration with graded pieces $\cong \mathbb{G}_{a}$ .

This finishes the proof of the lemma. Note that from the exact sequence, one also sees that all orbits in the stable orbit of $\tilde{T}$ appear in $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ , and the bijection just established is compatible with the reduction map $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}{\twoheadrightarrow}\mathfrak{g}(1)(k)$ that sends $\tilde{T}\mapsto T$ .◻

The main result in this subsection is to translate the following result from § 3.1.

Lemma 4.6. For $0\leqslant m\leqslant g$ ,

$$\begin{eqnarray}J(\tilde{T},f_{m})=\frac{1}{\#J_{T}[2](k)}(\#\tilde{X}_{T,m}(k)-\#\tilde{X}_{T,m-1}(k)).\end{eqnarray}$$

and

$$\begin{eqnarray}J^{\text{st}}(\tilde{T},f_{m})=\#X_{T,m}(k)-\#X_{T,m-1}(k).\end{eqnarray}$$

Proof. To ease notation we deal with the case $m>0$ . The proof applies to $m=0$ case with a little change in various places. By [Reference TsaiTsa15a, Theorem 2.1], for $\tilde{h}\in \tilde{G}(F)$ , $\text{Ad}(\tilde{h})\tilde{T}\in \tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}$ if and only if $\tilde{h}\in \tilde{G}(F)_{x}$ . In particular, the centralizer $\text{Stab}_{\tilde{G}}(\tilde{T})(F)\subset \tilde{G}(F)_{x}$ . Moreover, $\tilde{G}(F)_{\mathbf{x},1/2}$ acts trivially on $f_{m}$ since $f_{m}$ is locally constant by $\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}\subset \tilde{\mathfrak{g}}(F)_{y,-1/2}$ . The integral is thus essentially a sum over $\tilde{G}(F)_{x}/\tilde{G}(F)_{\mathbf{x},1/2}\cong O_{2g+1}(k)$ .

The measure of $\tilde{G}(F)_{\mathbf{x},1/2}$ is equal to that of $\tilde{\mathfrak{g}}(F)_{\mathbf{x},1/2}$ , which in Appendix A can be checked to be $q^{-(2g^{2}+g)/2}$ . The measure of $\text{Stab}_{\tilde{G}}(\tilde{T})(F)_{1/2}=\text{Stab}_{\tilde{G}}(\tilde{T})(F)\cap \tilde{G}(F)_{\mathbf{x},1/2}$ is $1$ . The image of $\text{Stab}_{\tilde{G}}(\tilde{T})(F)$ in $\text{O}_{2g+1}(k)$ is equal to $\text{Stab}_{\text{O}_{2g+1}(k)}(T)$ , which has order $2\#J_{T}[2](k)$ . Also $|D(\tilde{T})|=q^{2g^{2}+g}$ . We thus have

(4.1) $$\begin{eqnarray}J(\tilde{T},f_{m})=\frac{1}{2\#J_{T}[2](k)}\mathop{\sum }_{\bar{h}\in \text{O}_{2g+1}(k)}f_{m}(\text{Ad}(\bar{h})(T))=\frac{1}{\#J_{T}[2](k)}\mathop{\sum }_{\bar{h}\in \text{SO}_{2g+1}(k)}f_{m}(\text{Ad}(\bar{h})(T)),\end{eqnarray}$$

where $f_{m}$ in the right-hand side is understood as a function on $\tilde{\mathfrak{g}}(F)_{\mathbf{x},-1/2}/\tilde{\mathfrak{g}}(F)_{\mathbf{x},0}\cong \mathfrak{g}(1)$ .

We have $\unicode[STIX]{x1D70C}_{m}$ acts on $V$ (the standard representation of $G(0)\cong \text{SO}_{2g+1}$ ) by weights $2g-m,2g-m-2,\ldots ,m,m-1,\ldots ,-m,-m-2,\ldots ,-2g+m$ . Let $V_{g},\ldots ,V_{-g}\subset V$ be the one-dimensional subspace on which $\unicode[STIX]{x1D70C}_{m}$ acts by scalars with corresponding weights (in order). Since the quadratic form $\langle \cdot \,,\cdot \rangle$ on $V$ is preserved by $G(0)$ , we have $V_{j}\subset V_{i}^{\bot }$ unless $i=-j$ , i.e. unless their weights sum up to zero. Note that $\unicode[STIX]{x1D70C}_{m}$ acts on $N_{m}$ with weight $-2$ . This implies $N_{m}(V_{i})=V_{i-1}$ for $m<i\leqslant g$ and $-g<i\leqslant -m$ , and that $N_{m}(V_{i+1})=V_{i-1}$ for $-m<i<m$ .

We also write $V_{{\geqslant}n}\,:=\,\bigoplus _{n\leqslant i\leqslant g}V_{i}$ . From the description of $N_{m}$ above, one sees that if $\text{Ad}(h)T\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ for some $h\in G(0)(k)$ , then there exists $W^{g}\subset V\oplus k$ such that $\unicode[STIX]{x1D70B}_{1}(W^{g})=h^{-1}V_{{\geqslant}1}$ , and the flag $(0\subset h^{-1}V_{g}\subset h^{-1}V_{{\geqslant}g-1}\subset \cdots \subset h^{-1}V_{{\geqslant}2}\subset W^{g})$ is $m$ -exact (see the definition of $m$ -exactness in the paragraph before Theorem 3.4).

In fact, by the definition of $N_{m}$ , there exists $v_{1}\in V_{1}$ such that $\langle v_{1},N_{m}(v_{1})\rangle =1$ . With it $W^{g}$ can be given by either of the following two choices $W^{g}=h^{-1}V_{{\geqslant}2}+(h^{-1}.v_{1},\pm 1)$ , where $h^{-1}V_{{\geqslant}2}$ is a subspace of $V$ and thus of $V\oplus k$ .

Conversely, if there exists an $m$ -exact flag $0\subset W^{1}\subset \cdots \subset W^{g}\subset V\oplus k$ , then there is a unique right $B(0)(k)$ -coset in $G(0)(k)$ , say $B(0)(k)\cdot h$ , such that $W^{i}=h^{-1}V_{{\geqslant}g-i+1}$ for $1\leqslant i<g$ and $\unicode[STIX]{x1D70B}_{1}(W^{g})=h^{-1}V_{{\geqslant}1}$ . In this right coset, one checks from the definition of $m$ -exactness that there are exactly two right $U(0)(k)$ -cosets, say $U(0)(k)\cdot h$ , such that $\text{Ad}(h)T\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ .

In other words, there is a two-to-two correspondence between such $m$ -exact flags and right $U(0)(k)$ -cosets $U(0)(k)\cdot h\subset G(0)(k)$ satisfying $\text{Ad}(h)T\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ . Since $f_{m}(X)=q^{-g^{2}}$ when $X\in N_{m}+\mathfrak{g}(1)_{{\geqslant}-1}$ (recall $q^{-g^{2}}=(\#U(0)(k))^{-1}$ ), we conclude from (4.1) that

$$\begin{eqnarray}J(\tilde{T},f_{m})=\frac{1}{\#J_{T}[2](k)}\#F_{T,m}(k)=\frac{1}{\#J_{T}[2](k)}(\tilde{X}_{T,m}(k)-\tilde{X}_{T,m-1}(k)).\end{eqnarray}$$

The last equality follows from Theorem 3.4. This proves the first statement of the lemma.

For the stable Shalika germ, by Lemma 4.5 we have $T$ running over $G(0)(k)$ -orbits in its stable orbit, classified by $H^{1}(k,J_{T}[2])$ . When $T$ runs over these orbits, above every $k$ -point of $X_{T,m}$ , all isomorphism classes of $J_{T}[2]$ -torsor will appear exactly once in $\tilde{X}_{T,m}$ . Since $\#H^{1}(k,J_{T}[2])=\#H^{0}(k,J_{T}[2])=\#J_{T}[2](k)$ , the sum of the number of $k$ -points in all isomorphism classes is exactly $\#J_{T}[2](k)$ . This gives

$$\begin{eqnarray}J^{\text{st}}(\tilde{T},f_{m})=\#X_{T,m}(k)-\#X_{T,m-1}(k).\end{eqnarray}$$

◻

It is now straightforward to verify that Proposition 4.3 follows from Lemma 4.6 and the following lemma.

4.2 Nilpotent orbital integrals

We now prove Proposition 4.4. Nilpotent orbital integrals, just like counting points on nilpotent Springer fibers, is usually purely combinatorial. Let $B^{\text{opp}}\subset G$ be the opposite Borel to $B$ with respect to $S$ . Our essential idea here is that after applying the formula of Ranga Rao [Reference Ranga RaoRR72, Theorem 1], we can do ‘reduction modulo $\unicode[STIX]{x1D70B}$ ’ and arrive at an integral over $G(0)(k)$ which is left invariant by $U(0)(k)$ and right invariant by $B^{\text{opp}}(0)(k)$ . This gives a combinatorial sum over $U(0)(k)\backslash G(0)(k)/B^{\text{opp}}(0)(k)$ , which is identified with the Weyl group of $G(0)$ .

To begin our proof, the formula of Ranga Rao in our case can be formulated as follows. There exists a maximal $F$ -split torus $\tilde{S}\subset \tilde{G}$ , whose corresponding apartment contains $\mathbf{x}$ and whose reduction at $\mathbf{x}$ is equal to $S(0)\subset G(0)$ . Moreover, after conjugation we may assume that the cocharacter $\unicode[STIX]{x1D70C}_{m^{\prime }}\,:\,\mathbb{G}_{m}/_{k}\rightarrow S(0)$ corresponds to $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}\,:\,\mathbb{G}_{m}/_{F}\rightarrow \tilde{S}$ and that $\tilde{\unicode[STIX]{x1D70C}}_{m^{\prime }}$ also acts on ${\tilde{N}}_{m}^{\prime }$ by weight $-2$ .