2.1

For $g_0 \in G$ , we denote by $\varphi _{g_0}$ the automorphism of G given by

$$\begin{align*} \begin{array}{@{}ccccc} &\varphi_{g_0}:& G &\longrightarrow& G \\ & & g &\longmapsto& g_0gg_0^{-1}. \end{array} \end{align*}$$

The differential of $\varphi _{g_0}$ at the identity element induces an automorphism of the Lie algebra $ \mathfrak {g}$ denoted by $\mathrm {Ad}g_0$ . We have

$$ \begin{align*}g_0 \exp(X)g_0^{-1} = \exp(\mathrm{Ad}g_0.X), \, \, \forall X \in \mathfrak{g}.\end{align*} $$

The map $\mathrm {Ad}: G \longrightarrow \mathrm {GL}( \mathfrak {g})$ is a morphism of Lie groups, called the adjoint representation of G.

2.2

We designate by $ \mathfrak {g}^*$ the dual vector space of $ \mathfrak {g}$ . The group G operates on $ \mathfrak {g}^*$ by the coadjoint action:

$$ \begin{align*}\langle \mathrm{Ad}^*g.l,X \rangle = \langle l,\mathrm{Ad}g^{-1}.X \rangle,\, \, g\in G, l\in \mathfrak{g}^*, X\in \mathfrak{g}.\end{align*} $$

For $l\in \mathfrak {g}^*$ , we denote simply $g.l$ the coadjoint action of $g \in G$ on l. The set $ G.l =\left \{g.l, \, g\in G\right \}$ is called the coadjoint orbit of l in $ \mathfrak {g}^*$ . We denote by $G \backslash \mathfrak {g}^*$ the set of all coadjoint orbits.

Let $ l \in \mathfrak {g}^*$ . We designate by $\beta _l$ the skew-symmetric bilinear form on $ \mathfrak {g}$ defined by

(2.1) $$ \begin{align} \beta_l(X,Y ) = \langle l, [X,Y] \rangle, \, \, X,Y \in \mathfrak{g} \end{align} $$

and by $ \mathfrak {g}(l) = \{X \in \mathfrak {g} \, \vert \, \langle l, [X,Y] \rangle =0, \, \forall \, Y \in \mathfrak {g} \}$ its radical.

We have the following result.

The next result is of proper interest.

Proof Let

$$ \begin{align*}\mathfrak{g}_0 = \{0\} \subset \mathfrak{g}_1 \subset \cdots \subset \mathfrak{g}_n = \mathfrak{g}\end{align*} $$

be a non-decreasing sequence of ideals in $ \mathfrak {g}$ such that

1. — $\dim \mathfrak {g}_{i}=i$ and $[ \mathfrak {g}, \mathfrak {g}_{i}] \subset \mathfrak {g}_{i-1}$ , for all $i= 1, \ldots , n$ ;

2. — $ \mathfrak {g}_{\dim \mathfrak {h}}= \mathfrak {h} $ .

Put $\mathfrak {b} = \sum _{i=1}^n \mathfrak {g}_i(l_{\vert \mathfrak {g}_i})$ . From [Reference Bernat1, Chapter IV – Proposition 4.1.1], $ \mathfrak {b} $ is a polarization at l containing $\mathfrak {h}$ .

3.1

Let $f\in g^*$ and $\mathfrak {h}$ be a subordinate subalgebra to f. Let us denote by $H = \exp (\mathfrak {h})$ .

1. (1) For any $l \in f + \mathfrak {h}^\perp $ , the subalgebra $\mathfrak {h}$ is subordinate to l. Moreover, if $\mathfrak {h}$ is a polarization at $f,$ then is also one at l. Indeed, if $X, Y \in \mathfrak {h},$ then

   $$ \begin{align*}\beta_l(X,Y)= \langle l, [X, Y] \rangle = \langle f, [X, Y] \rangle = 0,\end{align*} $$
   since $ [X, Y] \in \mathfrak {h} $ .

   Now assume that $\mathfrak {h}$ is a polarization at f. Let $\chi _l$ be the character of H associated with l and $\pi _{l, \mathfrak {h}} = \mbox {Ind}_H^G \chi _l$ . Since l and f have the same restriction to $\mathfrak {h}$ , we have $\chi _l = \chi _f$ . So that $ \mbox {Ind}_H^G \chi _l = \mbox {Ind}_H^G \chi _f$ . Hence, $\pi _{l, \mathfrak {h}}$ is irreducible and thus $\mathfrak {h}$ is a polarization at l (see Theorem 4.2.1 below).

2. (2) The affine space $f + \mathfrak {h}^\perp $ is H-stable. Indeed, let $l \in f + \mathfrak {h}^\perp $ and $x = \exp (X) \in H$ . For any $Y \in \mathfrak {h}$ , we have

   $$ \begin{align*} \langle x.l, Y\rangle &= \langle l, \mathrm{Ad }x^{-1}.Y\rangle = \langle l, \exp(- \mathrm{ad}X).Y\rangle \\ &= \sum_{n=0}^{\dim \mathfrak{g}}\frac{(-1)^n}{n!} \langle l, (\mathrm{ad} X)^nY\rangle = \langle l, Y\rangle, \end{align*} $$
   since $ (\mathrm {ad} X)^nY \in [\mathfrak {h}, \mathfrak {h}]$ , for all $ n\in \mathbb N^*$ , and $\mathfrak {h}$ is subordinate to l. It follows that $ x.l-l \in \mathfrak {h}^\perp $ .

3. (3) The subspace $\mathfrak {h} + \mathfrak {g}(f)$ is totally isotropic for $\beta _f$ : Let $X,X' \in \mathfrak {h}, Y, Y' \in \mathfrak {g}(f)$ . We have

   $$ \begin{align*} \langle f, [X + Y, X'+ Y'] \rangle &= \langle f, [X,X']+ [X,Y'] + [Y, X'] + [Y, Y'] \rangle= 0. \end{align*} $$

4. (4) $ \mathfrak {g}.f = \mathfrak {g}(f)^\perp $ : This assertion follows from the fact $ \mathfrak {g}.f \subset \mathfrak {g}(f)^\perp $ and $ \dim \mathfrak {g}.f = \dim \mathfrak {g}(f)^\perp $ .

5. (5) $\mathfrak {h} + \mathfrak {g}(f) $ is a maximal isotropic subspace for $\beta _f$ if and only if $ \dim H.f=\frac {1}{2}\dim G.f$ . Indeed, we have

   $$ \begin{align*} \dim(\mathfrak{h} + \mathfrak{g}(f))&= \dim \mathfrak{h} +\dim \mathfrak{g}(f) - \dim(\mathfrak{h}\cap \mathfrak{g}(f))\\ &= \dim \mathfrak{h} +\dim \mathfrak{g}(f) - \dim\mathfrak{h}(f) = \dim H.f+\dim \mathfrak{g}(f). \end{align*} $$
   So that
   $$ \begin{align*} & \mathfrak{h} + \mathfrak{g}(f) \mbox{ is maximal isotropic subspace for }\beta_f \\ & \iff \dim(\mathfrak{h} + \mathfrak{g}(f)) = \frac{1}{2}(\dim \mathfrak{g} + \dim \mathfrak{g}(f)) \\ &\iff \dim H.f= \frac{1}{2}(\dim \mathfrak{g} - \dim \mathfrak{g}(f)) =\frac{1}{2}\dim G.f. \end{align*} $$

3.2

Let $f \in \mathfrak {g}^*$ . Denote by $S(f, \mathfrak {g})$ the set of all subordinate subalgebras to f and $Q(f, \mathfrak {g})$ the subset of subalgebras $\mathfrak {h}$ such that $\mathfrak {h} + \mathfrak {g}(f)$ is maximal isotropic subspace (for $\beta _f$ ).

Let $\mathfrak {h} \in S(f, \mathfrak {g})$ . In this paragraph, we investigate the intersection $ ( f + \mathfrak {h}^\perp ) \cap \Omega $ , where $\Omega \in G\backslash \mathfrak {g}^*$ is a coadjoint orbit. To this end, let

$$ \begin{align*} & \Xi= \{l \in f+ \mathfrak{h}^\perp \, \vert \, \mathfrak{h} \in Q(l, \mathfrak{g}) \}\\ & Z(f + \mathfrak{h}^\perp) = \{l \in f+ \mathfrak{h}^\perp \, \mbox{ such that }\dim \mathfrak{g}(l) \mbox{ is minimal when }l \mbox{ ranges over }f+ \mathfrak{h}^\perp\}. \end{align*} $$

Put $\Gamma = \Xi \cap Z(f + \mathfrak {h}^\perp ) $ .

Proof Let $l \in f + \mathfrak {h}^\perp $ . By point (1) of Section 3.1, the subalgebra $\mathfrak {h}$ is subordinate to l. By point (3), $\mathfrak {h} + \mathfrak {g}(f)$ is totally isotropic for $\beta _l$ . It follows that

$$ \begin{align*} \dim \mathfrak{h} +\dim \mathfrak{g}(l) - \dim(\mathfrak{h}\cap \mathfrak{g}(l)) & = \dim(\mathfrak{h} + \mathfrak{g}(l)) \leq\frac{1}{2}(\dim \mathfrak{g} + \dim \mathfrak{g}(l)). \end{align*} $$

So that

$$ \begin{align*}\dim(\mathfrak{h}\cap \mathfrak{g}(l)) \geq \dim\mathfrak{h} - \frac{1}{2}\dim G.l \geq r,\end{align*} $$

where $ r : =\dim \mathfrak {h} - \frac {1}{2}\dim G.l_0$ , $l_0 \in Z(f + \mathfrak {h}^\perp )$ . Then we have

$$ \begin{align*} l \in \Gamma &\iff \dim(\mathfrak{h} + \mathfrak{g}(l)) =\frac{1}{2}(\dim \mathfrak{g} + \dim \mathfrak{g}(l)) \mbox{ and } l \in Z(f + \mathfrak{h}^\perp) \\ & \iff \dim(\mathfrak{h}\cap \mathfrak{g}(l)) = \dim\mathfrak{h} - \frac{1}{2}\dim G.l = r. \end{align*} $$

But $\dim (\mathfrak {h}\cap \mathfrak {g}(l))>r$ means that $ \mathfrak {g}(l) \cap \mathfrak {h}\cap W \neq 0$ for every subspace W of $ \mathfrak {g}$ of dimension $\dim \mathfrak {g} -r$ . From [Reference Dixmier2, Lemma 1.11.4], for a given subspace $W \subset \mathfrak {g}$ , the set of $l \in \mathfrak {g}^*$ such that $ \mathfrak {g}(l) \cap \mathfrak {h}\cap W \neq 0$ is a closed subset of $ \mathfrak {g}^*$ . We deduce that the set of $l \in f + \mathfrak {h}^\perp $ such that $\dim (\mathfrak {h}\cap \mathfrak {g}(l))>r$ is closed in $ f + \mathfrak {h}^\perp $ and therefore its complement is open in $ f + \mathfrak {h}^\perp $ .

Proof Let $l' \in (f+ \mathfrak {h}^\perp )\cap G.l$ . We have

$$ \begin{align*} \dim ( \mathfrak{g}.l' \cap \mathfrak{h}^\perp )& = \dim( \mathfrak{g}(l')^\perp \cap \mathfrak{h}^\perp)=\dim(\mathfrak{h} + \mathfrak{g}(l'))^\perp = \dim \mathfrak{g} - \dim(\mathfrak{h} + \mathfrak{g}(l')) \\ & = \dim \mathfrak{g} -( \dim H.l' + \dim \mathfrak{g}(l'))=\dim G.l' - \dim H.l'. \end{align*} $$

Since $l' \in \Xi $ , it follows from (5) of Section 3.1 that $\dim H.l' = \frac {1}{2}\dim G.l'$ . Hence,

$$ \begin{align*}\dim ( \mathfrak{g}.l' \cap \mathfrak{h}^\perp ) = \dim H.l' = \dim \mathfrak{h}.l'.\end{align*} $$

Now,

$$ \begin{align*}\mathfrak{h}.l'= T_{l'}(H.l') \subset T_{l'}((f+ \mathfrak{h}^\perp)\cap G.l)\subset \mathfrak{h}^\perp\cap \mathfrak{g}.{l'}.\end{align*} $$

Using the equality of dimensions, this yields

$$ \begin{align*}T_{l'}(H.l') = T_{l'}((f+ \mathfrak{h}^\perp)\cap G.l).\end{align*} $$

Therefore, the tangent space to $(f+ \mathfrak {h}^\perp )\cap G.l$ at $l'$ has dimension $\frac {1}{2}\dim G.l$ , which is constant. It follows that every point of $(f+ \mathfrak {h}^\perp )\cap G.l$ is smooth.

Moreover, for each $l' \in (f+ \mathfrak {h}^\perp )\cap G.l$ , the orbit $H.l'$ is an open subvariety of $ (f+ \mathfrak {h}^\perp )\cap G.l$ . But $H.l'$ is an orbit under the action of a unipotent group, it is also a closed subset. As $H.l'$ is an irreducible set, it is an irreducible component of $ (f+ \mathfrak {h}^\perp ) \cap G.l$ . Thus, every irreducible component of $(f+ \mathfrak {h}^\perp )\cap G.l$ is a H-orbit of dimension $\frac {1}{2} \dim G.l$ .

Particular cases:

1. (1) Assume that $\mathfrak {h}$ is a polarization at f. It follows that $\mathfrak {h}$ is also a polarization at every element of $ f + \mathfrak {h}^\perp $ and so that $\Xi =f+ \mathfrak {h}^\perp = H.f= (f+ \mathfrak {h}^\perp )\cap G.l $ , for any $l \in f+ \mathfrak {h}^\perp $ .

2. (2) Assume that $ \mathfrak {h} $ is an ideal of $ \mathfrak {g}$ . For every $l \in \Xi $ , we have $(f+ \mathfrak {h}^\perp )\cap G.l = H.l$ . Indeed, set

   $$ \begin{align*}\mathfrak{h}^{\perp \beta_l} = \{X \in \mathfrak{g} \, \vert \, \langle l, [X, Y] \rangle = 0, \, \forall Y \in \mathfrak{h}\} \mbox{ and } G(l_{\vert \mathfrak{h}}) = \{x \in G \, \vert \, x.l_{\vert \mathfrak{h}} = l_{\vert \mathfrak{h}} \} .\end{align*} $$
   Then $ G(l_{\vert \mathfrak {h}})$ is an algebraic subgroup of G whose Lie algebra is $ \mathfrak {h}^{\perp \beta _l} $ . A straightforward computation gives
   $$ \begin{align*}(f+ \mathfrak{h}^\perp)\cap G.l = G(l_{\vert \mathfrak{h}}) .l.\end{align*} $$
   Next, observe that $ \mathfrak {h}^{\perp \beta _l} = (\mathfrak {h}+ \mathfrak {g}(l))^{\perp \beta _l} =\mathfrak {h}+ \mathfrak {g}(l) $ , since $\mathfrak {h}+ \mathfrak {g}(l)$ is a polarization at l. Therefore, $ G(l_{\vert \mathfrak {h}}) = HG(l) $ and hence
   $$ \begin{align*}(f+ \mathfrak{h}^\perp)\cap G.l = G(l_{\vert \mathfrak{h}}) .l = H.l \subset \Xi.\end{align*} $$

3. (3) Assume that $ \mathfrak {g}$ is a two-step nilpotent Lie algebra. This means that $[ \mathfrak {g}, \mathfrak {g}]$ is contained in the center of $ \mathfrak {g}$ . It follows that, for any $l \in \mathfrak {g}^*$ , we have $ [ \mathfrak {g}, \mathfrak {g}] \subset \mathfrak {g}(l) $ and thus $ \mathfrak {g}(l)$ is an ideal of $ \mathfrak {g}$ .

   Let $l \in \Xi $ . As $ \mathfrak {g}(x.l)= \mathfrak {g}(l)$ , then $\mathfrak {h} + \mathfrak {g}(x.l)$ is a polarization at $x.l$ , for any $x \in G$ . We deduce that $\Xi \supset (f+ \mathfrak {h}^\perp )\cap G.l$ .

4.1

Let X be a locally compact space such that each point has a fundamental system of open compact neighborhoods. We denote by $ \mathcal {C}_c^\infty (X)$ the space of locally constant $\phi : G \longrightarrow \mathbb C$ whose support $\mbox {Supp}(\phi )$ is compact. By a distribution on X, we mean a linear functional on $ \mathcal {C}_c^\infty (X)$ . We say that a distribution $\theta $ on X vanishes on an open subset $Y\subset X$ if

$$ \begin{align*}\langle \theta, \phi \rangle = 0, \mbox{ for all } \phi \in \mathcal{C}_c^\infty(X) \mbox{ with } \mbox{Supp}(\phi) \subset Y.\end{align*} $$

The support of $\theta $ , denoted $\mbox {Supp} (\theta )$ , is the set of points $x \in X$ such that $\theta $ does not vanish on any neighborhood of x.

4.2

A unitary representation of G in a Hilbert space $\mathcal {H}$ means a homomorphism $\pi $ of G into the group $U(\mathcal {H})$ of unitary operators on $\mathcal {H}$ such that, for every $v \in \mathcal {H},$ the map $G \ni x\longmapsto \pi (x).v \in \mathcal {H}$ is continuous.

An invariant subspace of $\pi $ means a subspace $\mathcal L$ of $\mathcal {H}$ satisfying $\pi (x) \mathcal L \subset \mathcal L$ , for all $x \in G$ . When there is no closed invariant subspace other than $\{0\}$ and $\mathcal {H}$ , $\pi $ is said to be irreducible.

Let $\rho $ be another unitary representation of G in a Hilbert space $\mathcal {H}_{\rho }$ . We say that $\pi $ and $\rho $ are equivalent if there is a unitary operator $T : \mathcal {H} \longrightarrow \mathcal {H}_{\rho }$ such that $T \pi (x)= \rho (x)T$ for every $x \in G$ . The set of all equivalence classes of irreducible unitary representations of G is denoted by $\widehat {G}$ .

Take now a closed subgroup K of G and let $\sigma $ be a unitary representation of K acting in a Hilbert space $\mathcal {H}_\sigma $ . We denote by $\mbox {Ind}_K^G\sigma $ the induced representation of G by $\sigma $ . The Hilbert space $\mathcal {H}$ of $\mbox {Ind}_K^G\sigma $ is the completion of the space:

$$ \begin{align*} \mathcal E(G/K, \sigma) = \left\{ \begin{matrix} \phi: G \longrightarrow \mathcal{H}_\sigma \mbox{ continuous with compact support modulo } K \, \\ \mbox{satisfying }\phi(gh) = \sigma(h^{-1}) \phi(g), \, \forall h\in K, \forall g \in G \end{matrix} \right\} \end{align*} $$

with respect to the norm

$$ \begin{align*}|| \varphi || = \left(\int_{G/K}||\varphi(g)||^2_{ \mathcal{H}_\sigma} \mathop{\mathrm{\mathstrut d}}\dot g \right)^{\frac{1}{2}},\end{align*} $$

where $\mathop {\mathrm {\mathstrut d}}\dot g $ is a G-invariant Borel measure on the homogeneous space $G/K$ . The group G acts on $\mathcal {H}$ by left translations, namely,

$$ \begin{align*}\mbox{Ind}_K^G\sigma(g) (\phi)(g') = \phi(g^{-1}g'), \, \mbox{ for } g,g' \mbox{ in } G, \, \phi \mbox{ in } \mathcal{H}.\end{align*} $$

Let $\mathfrak {b}$ be a subalgebra of $ \mathfrak {g}$ subordinate to $l\in \mathfrak {g}^*$ . Designate by $\chi _l$ the character of $B = \exp (\mathfrak {b})$ defined by

$$ \begin{align*} \chi_l(\exp(X)) = \psi(\langle l, X\rangle) , \, \, X \in \mathfrak{b}. \end{align*} $$

Denote by $\pi _{l,\mathfrak {b}}$ the induced representation of G: $ \pi _{l,\mathfrak {b}} = \mbox {Ind}_B^G \chi _l$ acting in the Hilbert space $\mathcal {H}_{l,\mathfrak {b}}$ .

This theorem is due to Kirillov [Reference Kirillov7] for the real case. Thereafter, Moore [Reference Moore12] has observed that Kirillov’s orbit method also applies to the group G. Recently, Martringe ([Reference Matringe10, Theorem 3.6] and [Reference Matringe11, Appendix]) gave a complete proof of the theorem for the smooth irreducible representations.

4.3

Let $\pi $ be a unitary representation of G acting on a Hilbert space $\mathcal {H}_{\pi }$ . A vector $v \in {\mathcal {H}}_{\pi }$ is said to be smooth with respect to $\pi $ if the map $G \ni g \mapsto \pi (g)v $ is locally constant. Let ${\mathcal {H}}_{\pi }^{\infty }$ be the set of all smooth vectors of $\pi $ . It is a G-invariant dense subspace of ${\mathcal {H}}_{\pi } $ . The dual of ${\mathcal {H}}^{\infty }_{\pi }$ is called the space of distributions of $\pi $ and is denoted by ${\mathcal {H}}_{\pi }^{-\infty }$ . By transposition, we define a representation $ \pi _{-\infty }$ of G into ${\mathcal {H}}_{\pi }^{-\infty }$ by the following formula:

$$ \begin{align*}\langle \pi_{-\infty}(g)a,v \rangle = \langle a, \pi(g^{-1})v \rangle, \, \, \forall a \in {\mathcal{H}}_{\pi}^{-\infty}, \forall v \in {\mathcal{H}}_{\pi}^{\infty} , \forall g \in G.\end{align*} $$

4.4

In this section, we assume that $\pi $ is irreducible and denote by $\Omega _\pi $ its corresponding coadjoint orbit. Let l be an element of $\Omega _\pi $ and $\mathfrak {b}$ be a polarization at l. Let us realize $\pi = \pi _{l, \mathfrak {b}}$ in the Hilbert space ${\mathcal {H}}_{l, \mathfrak {b}}$ .

4.4.1

Recall that an adapted complementary basis to $\mathfrak {b}$ in $ \mathfrak {g}$ is a basis $ (X_1, \ldots , X_m)$ of a subspace of $ \mathfrak {g}$ complementary to $\mathfrak {b}$ such that, for $ i = 0,\ldots , m-1$ , the subspace $ F X_{i+1} \oplus \dots \oplus F X_{m}\oplus \mathfrak {b}$ is a subalgebra of $ \mathfrak {g}$ . Since the Lie algebra $ \mathfrak {g}$ is nilpotent, such a basis exists (see [Reference Rais13, Chapter 5]). Putting $B = \exp (\mathfrak {b})$ , the map

$$\begin{align*} \begin{array}{@{}ccccc} \xi &:& F^m \times B&\longrightarrow& G\\ & &(t_1,\ldots, t_m, b) &\longmapsto& \exp(t_1X_1) \dots \exp(t_m X_m) b \end{array} \end{align*}$$

is a diffeomorphism from $F^m \times B$ onto G and the functional

$$ \begin{align*}\mathcal{C}_c^{\infty}(G/B) \ni \psi \longmapsto \int_{F^m} \psi(\exp(t_1X_1) \dots \exp(t_m X_m)B) dt_1 \dots dt_m\end{align*} $$

defines a G-invariant Borel measure on $G/B$ .

4.5

By the above analysis, the mapping

$$\begin{align*}\begin{array}{@{}ccccccc} \eta &:& {\mathcal{H}}_{l, \mathfrak{b}}&\longrightarrow& L^2(F^m)&&\\ & &\psi &\longmapsto& \eta ({\psi}):((t_1,\ldots, t_m)&\longmapsto & \psi (\exp(t_1X_1) \dots \exp(t_m X_m) )) \end{array} \end{align*}$$

is a bijective isometric map. Thus, we obtain an irreducible unitary representation $\sigma $ of G in the Hilbert space $L^2(F^m)$ , equivalent to $\pi $ :

$$ \begin{align*}\sigma(g) \eta ({\phi}) = \eta(\pi(g)\phi), \, \, \phi \in {\mathcal{H}}_{l, \mathfrak{b}}, \, g \in G .\end{align*} $$

Proposition 4.5.1 Keep notations and assumptions as above. The subspace $ \mathcal {C}_c^{\infty }(F^{m})$ of $ L^2(F^m)$ is stable by $\sigma $ and we have

$$ \begin{align*}(L^2(F^m) )^{\infty} = \mathcal{C}_c^{\infty}(F^{m}).\end{align*} $$

This is an analog of Kirillov’s result for the real case (see [Reference Howe6, Proposition 3.3]).

4.6

Let $\chi $ be a character of an algebraic subgroup $H $ of G. Recall that a distribution $a \in {\mathcal {H}}_{\pi }^{-\infty }$ is said to be $\chi $ -semi-invariant if it satisfies

$$ \begin{align*}\pi_{-\infty}(h)a = \chi(h^{-1})a, \, \forall h \in H.\end{align*} $$

We denote by $( {\mathcal {H}}_{\pi }^{-\infty })^{H, \chi }$ the space of all $\chi $ -semi-invariant distributions of $\pi $ . An interesting problem is the description of the space $( {\mathcal {H}}_{\pi }^{-\infty })^{H, \chi } $ .

Pick $f\in \mathfrak {g}^*$ such that $ \mathfrak {h} $ is subordinate to f and that $\chi = \chi _f$ . Assume that $ (f +\mathfrak {h}^\perp ) \cap \Omega _\pi \neq \emptyset $ . We claim that the space $(\mathcal {H}_{\pi }^{-\infty })^{H,\chi }$ is nonzero. For this, fix $g \in G$ such that $ g.l\in f +\mathfrak {h}^\perp $ .

Proposition 4.6.1 Keep the above notations. Let $d \dot {h}$ be a positive H-invariant measure on $H/H\cap gBg^{-1}$ . The linear form

(4.1) $$ \begin{align} I_{g.l, \pi}: {\mathcal{H}}_{l, \mathfrak{b}}^{\infty} \ni \phi \longmapsto\int_{H/H \cap gBg^{-1}} \phi(hg)\chi_f(h) \mathop{\mathrm{\mathstrut d}}\dot{h} \end{align} $$

defines a nonzero element of $ (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }$ .

Proof First, we show that the integrand is well-defined as a function on $H/H \cap gBg^{-1}$ , that is, it is constant on the cosets of $H \cap gBg^{-1}$ . Let $h \in H$ and $h' \in H \cap gBg^{-1}$ . We have

$$ \begin{align*} \phi(hh'g)\chi_f(hh')&= \phi(hg g^{-1}h'g)\chi_f(hh') = \chi_l(g^{-1}h^{\prime-1}g)\phi(hg)\chi_f(h)\chi_f(h')\\ &= \chi_{g.l}(h^{\prime-1})\chi_f(h')\phi(hg)\chi_f(h)= \phi(hg)\chi_f(h), \end{align*} $$

which proves the claim.

In the sequel, for simplicity, we assume that $g.l = f$ . Let $r_g$ be the map from G into itself defined by

$$ \begin{align*}r_g(x)= xg, \, x \in G.\end{align*} $$

Let $\phi \in \mathcal {H}_{l, \mathfrak {b}}^{\infty }$ , that is, $\phi : G\longrightarrow \mathbb C$ such that

1. (1) $\phi (xb) = \chi _l(b^{-1}) \phi (x)$ , $ b\in B, \, x \in G $ ;

2. (2) $\exists K$ an open compact subgroup of G such that $\phi (g_0x) = \phi (x), \, \forall x \in G, g_0 \in K$ ;

3. (3) $\mbox {Supp}( \phi )$ is compact modulo B.

Then the function $\phi \circ r_g$ is locally constant with compact support modulo $gBg^{-1}$ . From Lion–Perrin [Reference Lion and Perrin8], the integral

$$ \begin{align*} \int_{H/H \cap gBg^{-1}} \phi(hg)\chi_f(h) \mathop{\mathrm{\mathstrut d}}\dot{h} \end{align*} $$

converges.

Now we show that the linear form $I_{f, \pi }$ is nonzero. As $\phi \circ r_g \in \mathcal {H}_{g.l, g.\mathfrak {b}}^{\infty } $ , we may assume that $g =1$ . Assume that the adapted complementary basis $(X_1, \ldots , X_m) $ of $\mathfrak {b}$ in $ \mathfrak {g}$ is chosen such that $(X_{i_1}, \ldots , X_{i_p})$ is an adapted complementary basis of $\mathfrak {b} \cap \mathfrak {h}$ in $\mathfrak {h}$ . We consider the isometry map $\eta : \mathcal {H}_{l,\mathfrak {b}} \longrightarrow L^2(F^m)$ , introduced in Section 4.5. Pick an integer s large enough such that, $\mathcal {O}$ being the ring of integers of F,

$$ \begin{align*}\langle f, \varpi^s X_{i_r} \rangle \in \mathcal O, \, \forall r = 1, \ldots,p.\end{align*} $$

Put $\tilde {\beta }= 1_{\varpi ^s \mathcal O \times \cdots \times \varpi ^s \mathcal O } \in \mathcal {C}_c^{\infty }(F^m)$ . Let $\beta = \eta ^{-1}(\tilde {\beta })$ . We have $\beta \in \mathcal {H}_{l, \mathfrak {b}}^{\infty }$ . We show that $I_{f, \pi }(\beta ) \neq 0$ . As the map

$$ \begin{align*}\begin{array}{@{}ccccc} & &F^p \times (H \cap B) &\longrightarrow& H\\ & &(t_1,\ldots, t_p, b) &\longmapsto& \exp(t_1X_{i_1}) \dots \exp(t_p X_{i_p}) b \end{array}\end{align*} $$

is a diffeomorphism, for each $ h \in H$ , we may write

$$ \begin{align*}h = \exp(t_1X_{i_1}) \dots \exp(t_p X_{i_p}) b, \,\quad t_1, \ldots,t_p \in F, b \in H\cap B.\end{align*} $$

We deduce that

$$ \begin{align*} \beta(h)\chi_f(h) = \left\{ \begin{array}{@{}lr} 1 \quad \mbox{ if }\,(t_1,\ldots, t_p) \in \varpi^s \mathcal O \times \cdots \times \varpi^s \mathcal O \\ 0 \quad \mbox{otherwise}. \end{array} \right. \end{align*} $$

So that the function $h \mapsto \beta (h)\chi _f(h) $ is nonnegative and nonzero. It follows that

$$ \begin{align*}\int_{H/H \cap B}\beta(h)\chi_f(h)\mathop{\mathrm{\mathstrut d}}\dot{h}>0.\end{align*} $$

For the $\chi $ -semi-invariance of $I_{f, \pi }$ , let $h_0 \in H$ and $\phi \in \mathcal {H}_{l, \mathfrak {b}}^{\infty }$ :

$$ \begin{align*} \langle \pi_{-\infty}(h_0) I_{f, \pi}, \phi \rangle &= \langle I_{f, \pi}, \pi(h_0^{-1}) \phi \rangle \\ &= \int_{H/H \cap gBg^{-1}} \pi(h_0^{-1})\phi(hg)\chi_f(h)\mathop{\mathrm{\mathstrut d}}\dot{h} \\ &= \int_{H/H \cap gBg^{-1}} \phi(h_0hg)\chi_f(h)\mathop{\mathrm{\mathstrut d}}\dot{h} \\ &= \int_{H/H \cap gBg^{-1}} \phi(hg)\chi_f(h_0^{-1}h)\mathop{\mathrm{\mathstrut d}}\dot{h} \\ &= \chi_f(h_0^{-1}) \int_{H/H \cap gBg^{-1}} \phi(hg)\chi_f(h)\mathop{\mathrm{\mathstrut d}}\dot{h} \\ &= \chi_f(h_0^{-1}) \langle I_{f, \pi}, \phi \rangle= \langle \chi_f(h_0^{-1}) I_{f, \pi}, \phi \rangle. \end{align*} $$

Thus, the desired result.

Proof Set $K= gBg^{-1}$ . The map

$$ \begin{align*}H/H\cap K \longrightarrow H/H \cap h_0Kh_0^{-1}, \, h \mod H\cap K \longmapsto h_0hh_0^{-1} \mod H\cap h_0K h_0^{-1}\end{align*} $$

is a diffeomorphism and sends a positive H-invariant measure on $H/H\cap K$ to a positive H-invariant measure on $H/H\cap h_0K h_0^{-1}$ . So that we have

$$ \begin{align*} \langle I_{h_0f, \pi}, \phi \rangle &= \int_{H/H \cap h_0K h_0^{-1}} \phi(hh_0g)\chi_{h_0f}(h){\mathrm{\mathstrut d}}_{H/H \cap h_0K h_0^{-1}}\dot{h}\\ &= c\int_{H/H \cap K } \phi(h_0hg)\chi_{h_0f}(h_0hh_0^{-1}){\mathrm{\mathstrut d}}_{H/H \cap K }\dot{h}\\ &= c \chi_{f}(h_0^{-1}) \langle I_{f, \pi}, \phi \rangle , \end{align*} $$

where c is some constant. Thus, the result.

4.7

Keep the notations of Section 3.2 and the following ones. Here, we will compute the space $ (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }$ , for $l \in \Xi $ such that $\Xi \supset (f + \mathfrak {h}^\perp )\cap G.l$ . By Proposition 3.2.1, $(f + \mathfrak {h}^\perp ) \cap G.l$ is a smooth variety and every irreducible component of $ (f + \mathfrak {h}^\perp )\cap G.l $ is a H-orbit of dimension $\frac {1}{2} \dim G.l$ .

Let us denote by $C_1, \ldots , C_n$ the irreducible components of $(f + \mathfrak {h}^\perp ) \cap G.l$ . For each $1 \leq j \leq n$ , pick $g_j \in G$ such that $g_j.l \in C_j$ .

Let $\mathfrak {b}$ be a fixed polarization at l and $B = \exp (\mathfrak {b})$ . Let $\mathop {\mathrm {\mathstrut d}}{b}$ be a Haar measure on B.

For each $1 \leq j \leq n$ , we consider the distribution $I_{g_j.l, \pi }$ defined by formula (4.1).

For $\phi \in \mathcal {C}_c^\infty (G)$ , we denote by $\tilde \phi $ the function on G defined by

$$ \begin{align*}\tilde \phi (x)= \int_B\phi(xb) \chi_l(b)\mathop{\mathrm{\mathstrut d}}{b}, \, x \in G.\end{align*} $$

We see immediately that $\tilde \phi \in \mathcal {H}_{l, \mathfrak {b}}^{\infty }$ . So that if $a \in \mathcal {H}_{l, \mathfrak {b}}^{-\infty }$ , we obtain a distribution $\tilde a$ on G by putting

$$ \begin{align*}\langle \tilde a, \phi \rangle = \langle a, \tilde \phi \rangle , \, \phi \in \mathcal{C}_c^\infty(G).\end{align*} $$

The support of $ \widetilde {I_{g_j.l, \pi }} $ is contained in the closed subset $Hg_jB$ . Indeed, let $\phi \in \mathcal {C}_c^\infty (G)$ such that $\mbox {Supp}(\phi ) \subset G \setminus Hg_jB$ . Then $\tilde \phi (hg_j) = 0 $ , for all $h \in H$ . Hence, $ \langle I_{g_j.l, \pi }, \tilde \phi \rangle = 0 $ .

Consequently, $I_{g_1.l, \pi }, \ldots , I_{g_n.l, \pi }$ are linearly independent.

Proof Take $g_j=1$ for simplicity.

For $\phi \in \mathcal {C}_c^\infty (G)$ , we define a function $\varphi _\phi $ on $H/(H\cap B)$ by putting

$$ \begin{align*}\varphi_\phi(h) = \tilde \phi(h)\chi_f(h), \, h \in H.\end{align*} $$

We have $ \varphi _\phi \in \mathcal {C}_c^\infty (H/(H\cap B))$ . This allows us to define a distribution $\theta $ on $H/(H\cap B)$ by

$$ \begin{align*}\langle \theta , \varphi_\phi \rangle =\langle a , \tilde \phi \rangle.\end{align*} $$

This is well-defined. Indeed, let $\phi _1 , \phi _2 \in \mathcal {C}_c^\infty (G)$ such that $ \varphi _{\phi _1}= \varphi _{\phi _2}$ . We have

$$ \begin{align*}\widetilde{\phi_1-\phi_2}(h)=0 , \forall h \in H.\end{align*} $$

So that $\widetilde {\phi _1-\phi _2}=0$ on $HB$ . Now let K be a compact open subgroup of G such that $\phi _1$ and $\phi _2$ are left K-invariant. A straightforward computation shows that

$$ \begin{align*}\widetilde{\phi_1-\phi_2} = \widetilde{{\textit{1}_{(KHB)^c}(\phi_1-\phi_2)}} ,\end{align*} $$

where $(KHB)^c = G \setminus KHB$ . Hence, we have

$$ \begin{align*}\langle a, \tilde \phi_1- \tilde\phi_2 \rangle = \langle a, \widetilde{\phi_1-\phi_2} \rangle = \langle a, \widetilde{{\textit{1}_{(KHB)^c}(\phi_1-\phi_2)}} \rangle = \langle \tilde a, 1_{(KHB)^c}(\phi_1-\phi_2) \rangle = 0.\end{align*} $$

Furthermore, for all $\alpha \in \mathcal {C}_c^\infty (H/(H\cap B))$ , there exists $\phi \in \mathcal {C}_c^\infty (G)$ such that $\alpha = \varphi _\phi $ (see Section 4.8 below).

The group H acts by left translation on $H/(H\cap B)$ . It induces an action on $ \mathcal {C}_c^\infty (H/(H\cap B))$ . For this action, the distribution $\theta $ is H-invariant. Indeed, for $h_0 \in H$ , we have

$$ \begin{align*} L_{h_0} \varphi_\phi(h) = \varphi_\phi(h_0^{-1}h) = \tilde \phi(h_0^{-1}h)\chi_f(h_0^{-1}h)=\widetilde{L_{h_0}\phi}(h)\chi_f(h) \chi_f(h_0^{-1}), \end{align*} $$

where $ L_{h_0} \phi (x)= \phi (h_0^{-1}x), x \in G $ . Thus,

$$ \begin{align*} \chi_f(h_0)L_{h_0}\varphi_\phi(h) =\widetilde{L_{h_0}\phi}(h)\chi_f(h)= \varphi_{L_{h_0}\phi}(h), \, h \in H, \end{align*} $$

Consequently,

$$ \begin{align*} \chi_f(h_0)\langle \theta ,L_{h_0} \varphi_\phi \rangle= \langle \theta , \varphi_{L_{h_0}\phi} \rangle= \langle a , \widetilde{L_{h_0}\phi} \rangle = \langle a , \pi(h_0)\widetilde{\phi}\rangle = \chi_f(h_0)\langle a , \widetilde{\phi}\rangle. \end{align*} $$

Hence, $ \langle \theta , L_{h_0} \varphi _\phi \rangle = \langle \theta , \varphi _\phi \rangle $ , which proves our claim.

Using [Reference Harish-Chandra5, Part V – Lemma 17], we see that there exists a constant $c \in \mathbb C$ such that

$$ \begin{align*}\langle \theta , \alpha \rangle = c \int_{H/H\cap B} \alpha(\dot{h}) \mathop{\mathrm{\mathstrut d}}\dot{h}, \, \, \alpha \in \mathcal{C}_c^\infty(H/(H\cap B)).\end{align*} $$

Thus, the result.

4.8

Pick an adapted supplementary basis $(X_1,\ldots ,X_m)$ of $\mathfrak {b}$ in $ \mathfrak {g}$ such that $(X_{i_1},\ldots ,X_{i_s})$ is an adapted supplementary basis of $\mathfrak {h} \cap \mathfrak {b}$ in $\mathfrak {b}$ . Consider the map

$$\begin{align*} \begin{array}{@{}ccccc} \xi' &:& F^s \times H\cap B&\longrightarrow& H\\ & &(t_1,\ldots, t_s, b) &\longmapsto& \exp(t_1X_{i_1}) \dots \exp(t_s X_{s}) b. \end{array} \end{align*}$$

This map is a diffeomorphism. It induces a diffeomorphism

$$ \begin{align*}\dot{\xi'} : F^s \longrightarrow H/ (H\cap B), \, \dot{\xi'}(t)= \xi'(t,e). H\cap B,\end{align*} $$

where e is the identity element of $H\cap B$ . Consequently, the map

$$ \begin{align*}\mathcal{C}^{\infty}_{c}(F^{s})\ni \alpha\mapsto\alpha \circ \dot{\xi'}^{-1} \in \mathcal{C}^{\infty}_{c}(H/H\cap B)\end{align*} $$

is a linear isomorphism.

Define a function $\chi _f' $ on $F^s$ by

$$ \begin{align*}\chi_f'(t_1,\ldots, t_s)= \chi_f(\exp(t_1X_{i_1}) \dots \exp(t_s X_{s})) .\end{align*} $$

Let $\alpha \in \mathcal {C}_c^\infty (F^s)$ . Choose $\beta _1 \in \mathcal {C}_c^\infty (F^m)$ such that $\beta _{1 \vert F^s}= \frac {\alpha }{\chi _f'}$ and $\beta _2 \in \mathcal {C}_c^\infty (B)$ such that $ \int _B\beta _2(b) \chi _l(b)db= 1$ . Set $\phi = (\beta _1 \otimes \beta _2)\circ \xi ^{-1}$ . Then $ \phi \in \mathcal {C}_c^\infty (G)$ and

$$ \begin{align*}\alpha \circ \dot{\xi'}^{-1} (\dot{h})= \tilde \phi(h) \chi_f(h), \, \forall h \in H.\end{align*} $$

4.9

Put

$$ \begin{align*}S_\pi= \{g \in G \, \vert \, g.(l+ \mathfrak{b}^\perp)\cap(f+ \mathfrak{h}^\perp) \neq \emptyset\}.\end{align*} $$

Using the following facts:

1. — The intersection of two affine subspaces is an affine subspace or the emptyset,

2. — each affine subspace is an irreducible subset,

3. — the irreducible components of $(f+ \mathfrak {h}^\perp )\cap G.l$ are $C_1 = Hg_1.l, \ldots , C_n = Hg_n.l$ ,

we show that $S_\pi = \sqcup _{k=1}^n Hg_kB$ . Indeed, let $g \in S_\pi $ . Since $B.l= l+ \mathfrak {b}^\perp $ , there exist $b \in B$ and $k \in {[\![} 1, n {]\!]}$ such that $ gb.l \in C_k= Hg_k.l$ . Hence, $g \in Hg_kB$ . Conversely, for any $h \in H$ and $b \in B$ , we have

$$ \begin{align*}hg_k.l \in hg_kb.(l+ \mathfrak{b}^\perp)\cap(f+ \mathfrak{h}^\perp).\end{align*} $$

So that $Hg_kB \subset S_\pi $ . Finally, if $j \neq k$ , then $ Hg_jB \neq Hg_kB$ . Indeed, otherwise, the irreducible subset $ g_k.(l+ \mathfrak {b}^\perp )\cap (f+ \mathfrak {h}^\perp )$ of $(f + \mathfrak {h}^\perp ) \cap G.l$ would intersect both $C_j$ and $C_k$ , which is impossible.

Denote by $(\mathcal {H}_{l, \mathfrak {b}}^{-\infty })_Z^{H,\chi }$ the subspace of $(\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }$ consisting of elements $a $ such that $\tilde a$ is supported by $ S_\pi $ .

Theorem 4.9.1 Let $\pi \in \widehat {G}$ be such that $(f+ \mathfrak {h}^\perp ) \cap \Omega _\pi \subset \Xi $ . Let $l\in \Omega _\pi $ and let $\mathfrak {b}$ be a polarization at l. We realize $\pi = \pi _{l, \mathfrak {b}}$ in the Hilbert space ${\mathcal {H}}_{l, \mathfrak {b}}$ . For each H-orbit $C_j$ in $(f + \mathfrak {h}^\perp ) \cap G.l$ , choose an element $g_j \in G$ such that $g_j.l \in C_j$ .

Then we have

$$ \begin{align*}(\mathcal{H}_{l, \mathfrak{b}}^{-\infty})_Z^{H,\chi} = \mathrm{Span}\{I_{g_1.l, \pi}, \ldots, I_{g_n.l, \pi}\},\end{align*} $$

and $\dim (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })_Z^{H,\chi }$ is equal to the number of H-orbits contained in $( f + \mathfrak {h}^\perp )\cap G.l$ .

Proof The only remaining point to prove is the direct inclusion. Let $a \in (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })_Z^{H,\chi }$ and $\phi \in \mathcal {C}_c^\infty (G)$ . We show that for each $k \in {[\![} 1, n rbracket$ , there exists $\phi _k \in \mathcal {C}_c^\infty (G)$ satisfying $ \begin {cases} \phi _{k \vert Hg_kB} = \phi _{\vert Hg_kB} \\ \phi _{k \vert \cup _{l=1; \, l\neq k}^n Hg_lB} =0. \end {cases}$ Set ${}^{11}A_k= \{x \in Hg_kB \, \vert \, \phi (x ) \neq 0 \}$ . Since $Hg_kB$ is closed in G, the set $A_k$ is a compact subset of $Hg_kB$ . Hence, there exists a compact open subset $M_k\subset G$ such that

$$ \begin{align*}A_k \subset M_k \quad \mbox{and} \quad (\cup_{l=1; \, l\neq k}^{l=n} Hg_lB) \cap M_k = \emptyset.\end{align*} $$

Then the function $ 1_{M_k}\phi $ satisfies the required conditions.

Define $ \langle a_k, \tilde \phi \rangle = \langle a, \tilde \phi _k \rangle. $ It is immediate that $a_k \in \mathcal {H}_{l, \mathfrak {b}}^{-\infty }$ and that $\mbox {Supp} (\tilde {a_k}) \subset Hg_kB$ . We claim that $a_k \in (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }$ . Indeed, let $h_0 \in H$ . Since $ \begin {cases} L_{h_0^{-1}}\phi _{k \vert Hg_kB} = L_{h_0^{-1}}\phi _{\vert Hg_kB} \\ L_{h_0^{-1}}\phi _{k \vert \cup _{l=1; \, l\neq k}^n Hg_lB} =0, \end {cases}$ we deduce that

$$ \begin{align*} \langle \pi_{-\infty}(h_0) a_k, \tilde \phi \rangle& = \langle a_k,\pi(h_0^{-1}) \tilde \phi \rangle =\langle a_k,\widetilde{L_{h_0^{-1}}\phi} \rangle = \langle a, \widetilde{L_{h_0^{-1}}\phi_k} \rangle = \langle a, \pi(h_0^{-1}) \tilde \phi_k \rangle \\ &= \langle \pi_{-\infty}(h_0) a, \tilde \phi_k \rangle = \chi(h_0^{-1}) \langle a, \tilde \phi_k \rangle = \chi(h_0^{-1}) \langle a_k, \tilde \phi \rangle = \langle\chi(h_0^{-1}) a_k, \tilde \phi \rangle. \end{align*} $$

Thus, $\pi _{-\infty }(h_0) a_k= \chi (h_0^{-1}) a_k$ and hence $a_k \in (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }$ .

By Proposition 4.7.1, there exists $c_k \in \mathbb C$ such that $ a_k = c_kI_{g_k.l, \pi }$ .

Finally, since $ ( \phi - \sum _{k=1}^n \phi _k )_{\vert S_\pi } = 0 $ , we obtain

$$ \begin{align*}\langle a, \tilde \phi \rangle = \langle a,\widetilde{ \sum_{k=1}^n \phi_k} \rangle= \langle a, \sum_{k=1}^n \tilde \phi_k \rangle=\sum_{k=1}^n \langle a, \tilde \phi_k \rangle = \sum_{k=1}^n \langle a_k, \tilde \phi \rangle =\langle \sum_{k=1}^n a_k, \tilde \phi \rangle .\end{align*} $$

Whence, $a= \sum _{k=1}^n a_k $ .

5.1

As H is normal in G, we have

$$ \begin{align*}gHg^{-1} \subset B, \, \forall g \in G.\end{align*} $$

Pick $a \in (\mathcal {H}_{\pi }^{-\infty })^{H,\chi _{f}}$ , $\phi \in \mathcal {H}_{\pi }^{\infty }$ , and $ h \in H$ . For each $g \in G$ , we have

$$ \begin{align*} \pi_\infty(h^{-1}) \phi(g ) &= \phi(hg) = \phi(gg^{-1}hg) = \chi_l(g^{-1}h^{-1}g)\phi(g). \end{align*} $$

So that

$$ \begin{align*} \langle \pi_{-\infty}(h) a , \phi \rangle &=\langle a , \pi_\infty(h^{-1})\phi \rangle = \langle a, \chi_l(g^{-1}h^{-1} g) \phi(g) \rangle. \end{align*} $$

On the other hand, a is $\chi _{f}$ -semi-invariant:

$$ \begin{align*} \langle \pi_{-\infty}(h) a , \phi \rangle &= \langle \chi_{f}(h^{-1}) a , \phi \rangle = \langle a , \chi_f(h^{-1}) \phi \rangle. \end{align*} $$

Hence, a satisfies the following condition:

$$ \begin{align*} \langle a, (g\longmapsto (\chi_f(h)- \chi_l(g^{-1}h g) ) \phi(g) ) \rangle = 0 , \quad \forall \phi \in \mathcal{H}_{\pi}^{\infty}, h \in H. \end{align*} $$

As before, let $ (X_1, \ldots , X_m)$ be an adapted supplementary basis of $\mathfrak {b}$ in $ \mathfrak {g}$ . We recall that the map $\begin {array}{ccccc} \xi &:& F^m \times B&\longrightarrow & G\\ & &(t_1,\ldots , t_m, b) &\longmapsto & \exp (t_1X_1) \dots \exp (t_m X_m) b \end {array}$ is a diffeomorphism. For $t = (t_1,\ldots , t_m) \in F^m$ , write $g_t = \exp (t_1X_1) \dots \exp (t_m X_m)$ . For $g \in G$ , we may write $g = g_t b \in G$ , where $b \in B$ . We have

$$ \begin{align*} (\chi_f(h)- \chi_l(g^{-1}h g) ) \phi(g) &= (\chi_f(h)- \chi_l(g_t^{-1}h g_t) ) \phi(g_t.b) \\ & = (\chi_f(h)- \chi_l(g_t^{-1}h g_t) ) \chi_l(b^{-1}) \phi(g_t) \\ & = \chi_l(b^{-1}) (\chi_f(h)- \chi_l(g_t^{-1}h g_t) ) \phi(g_t). \end{align*} $$

So a, considered as a distribution of $F^m$ , satisfies the hypothesis of the following lemma.

Lemma 5.1.1 Let $ b : \mathcal {C}_c^{\infty }(F^m) \longrightarrow \mathbb C$ be a linear map such that

(5.1) $$ \begin{align} \langle b , (\chi_f(h)- \chi_l(g_t^{-1}h g_t) ) \phi(t_1, \ldots , t_m) \rangle = 0 , \, \forall \phi \in \mathcal{C}_c^{\infty}(F^m), \forall h \in H. \end{align} $$

Then we have

$$ \begin{align*}\mbox{Supp} (b) \subset \left\{(t_1, \ldots, t_m)\in F^m, \exp(t_1 X_1) \dots \exp(t_m X_m).l \in f + \mathfrak{h}^{\perp}\right\}.\end{align*} $$

Proof Let $ S: = \left \{(t_1, \ldots , t_m)\in F^m, \exp (t_1 X_1) \dots \exp (t_m X_m).l \in f + \mathfrak {h}^{\perp }\right \}$ . Consider the following map:

$$\begin{align*} \begin{array}{@{}ccccc} \theta &:& F^m \times H &\longrightarrow& \mathbb C\\ & &((t_1, \ldots,t_m),h) &\longmapsto& \chi_f(h)- \chi_l(g_t^{-1}h g_t) = \chi_{g_t.l}(h)(\chi_{f-g_t.l}(h)-1), \end{array} \end{align*}$$

where $g_t = \exp (t_1 X_1) \dots \exp (t_m X_m)$ . For $t = (t_1, \ldots ,t_m) \in F^m$ , we have

$$ \begin{align*} \theta ((t_1, \ldots,t_m),h) =0 , \forall h \in H &\iff \chi_{f-g_t.l}(h) = 1 , \forall h \in H \\ &\iff f-g_t.l \in \mathfrak{h}^\perp \iff t \in S. \end{align*} $$

Now let $t = (t_1, \ldots ,t_m) \in F^m$ such that $ t \notin S$ . From the above, there exists $h_0 \in H$ such that $\theta (t, h_0) \neq 0$ . As $\theta $ is locally constant, there exists a neighborhood $V_t$ of t in $F^m$ such that

$$ \begin{align*}\theta(t', h_0) = \theta(t, h_0), \forall t' \in V_t.\end{align*} $$

It follows that, for all $\phi \in \mathcal {C}_c^\infty (F^m)$ such that $\mbox {Supp}(\phi ) \subset V_t$ , we have $\langle b , \phi \rangle =0$ . Hence, $\mbox {Supp}(b) \cap V_t = \emptyset $ .

5.2

We deduce from the above lemma that

$$ \begin{align*}\mbox{Supp}(a) \subset \left\{t\in F^m\vert \, g_t.l \in f + \mathfrak{h}^{\perp}\right\}.\end{align*} $$

By using the equality $ (f + \mathfrak {h}^{\perp })\cap G.l = H.l$ (see the second particular case in Section 3.2), we deduce that $ \mbox {Supp}(a) \subset \left \{0\right \}$ .

Let $a_0$ be the linear form on $ \mathcal {C}_c^\infty (F^m)$ defined by

$$ \begin{align*}\langle a_0, \phi \rangle = \phi(0), \, \phi \in \mathcal{C}_c^\infty(F^m).\end{align*} $$

By direct computation, we check that $a_0 \in (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }$ and $\mbox {Supp}(a_0) = \{0\} $ .

Now fix an open compact subset V of $F^m$ containing $0$ . Put $\alpha = \langle a, 1_V \rangle $ . We show that $ a = \alpha a_0$ .

We start with the following remark: for all open compact subset $V'$ of $F^m$ containing $0$ , we have $ \langle a, 1_V \rangle = \langle a, 1_{V'} \rangle $ , because $ 1_V- 1_{V'} $ is zero on $V \cap V'$ .

Let $\phi \in \mathcal {C}_c^\infty (F^m )$ such that $\phi (0) \neq 0$ . Put $V_{0} = \phi ^{-1}(\phi (0)),$ which is an open compact subset of $F^m$ . Writing $ \phi = (\phi - \phi (0)1_{V_{0}} ) + \phi (0)1_{V_{0}} $ , we obtain

$$ \begin{align*}\langle a, \phi \rangle = \langle a, 1_{V_{0}} \rangle \phi(0) =\alpha a_0(\phi).\end{align*} $$

So that $a = \alpha a_0$ .

Proposition 5.2.1 We have

$$ \begin{align*}(\mathcal{H}_{l, \mathfrak{b}}^{-\infty})^{H,\chi} = \mathbb C a_0.\end{align*} $$

Consequently, we have $ (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })^{H,\chi }= (\mathcal {H}_{l, \mathfrak {b}}^{-\infty })_Z^{H,\chi }$ .

7.1

In the following, we will describe the induced representation $\pi $ .

Let $\mathfrak {r}\subset \mathfrak {g}$ be a subspace such that $\mathfrak {g}=\mathfrak {r}\oplus \mathfrak {b}$ . Then the map: $ \mathfrak {r} \longrightarrow G/B, \, X\longmapsto \overline {\exp (X)}$ , is a diffeomorphism and a Haar measure $d_{\mathfrak {r}} X$ on $\mathfrak {r}$ defines a G-invariant positive measure $d\dot g$ on $G/B$ . Let $U_l : L^2(\mathfrak {r}, d_{\mathfrak {r}} X) \longrightarrow \mathcal {H}_{l,\mathfrak {b}}$ be the operator defined by

$$ \begin{align*} U_l(\varphi)(\exp(X)\exp(Y))= \chi_l(\exp(-Y))\varphi(X), \, Y \in B, X \in \mathfrak{r}, \varphi \in L^2(\mathfrak{r}, d_{\mathfrak{r}} X). \end{align*} $$

Then $U_l$ is unitary, and we obtain an irreducible unitary representation $\sigma _{l}$ of G acting on the Hilbert space $L^2(\mathfrak {r}, d_{\mathfrak {r}} X)$ :

$$ \begin{align*} \sigma_{l} (g)= U_l^{-1} \circ \pi_{l,\mathfrak{b}}(g) \circ U_l , \, \, g \in G. \end{align*} $$

Explicitly, we have

$$ \begin{align*} \sigma_{l} (\exp(X)\exp(Y))\varphi(X')= \psi(\langle l, Y+ [X-X', Y] + \frac{1}{2}[X, X'])\varphi(X'-X), \end{align*} $$

where $X,X' \in \mathfrak {r}, Y \in \mathfrak {b}, $ and $ \varphi \in L^2(\mathfrak {r}, d_{\mathfrak {r}} X)$ .

7.2

We have $ (L^2(\mathfrak {r}, d_{\mathfrak {r}} X))^\infty = \mathcal {C}_c^\infty (\mathfrak {r})$ .

Let $a \in (\mathcal {H}_{\pi }^{-\infty })^{H,\chi _{f}}$ . For $\phi \in \mathcal {C}_c^{\infty }(\mathfrak {r})$ and $ h = \exp (Y), \, Y \in \mathfrak {h}$ , we have

$$ \begin{align*}\sigma_{l}(h^{-1}) \phi(X') = \psi(\langle l, -Y + [X', Y])\phi(X'), \, X' \in \mathfrak{r}. \end{align*} $$

So that

$$ \begin{align*}\langle \sigma_{l, -\infty}(h) a , \phi \rangle = \langle a, (X'\longmapsto \psi(\langle l, -Y + [X', Y])\phi(X') )\rangle.\end{align*} $$

On the other hand, a is $\chi _{f}$ -semi-invariant:

$$ \begin{align*}\langle \sigma_{l, -\infty}(h) a , \phi \rangle = \langle \chi_{f}(h^{-1}) a , \phi \rangle = \langle a , \psi(\langle l, -Y \rangle) \phi \rangle. \end{align*} $$

Hence, a satisfies the following condition:

$$ \begin{align*}\langle a, (X'\longmapsto \psi(\langle l, [X', Y]\rangle )\phi(X') )\rangle =0, \forall \phi \in \mathcal{C}_c^\infty(\mathfrak{r}), \forall Y \in \mathfrak{h}.\end{align*} $$

We deduce that

$$ \begin{align*}\mbox{Supp}(a) \subset \{X' \in \mathfrak{r} \, \vert \langle l, [X', Y]\rangle = 0 ,\, \forall Y \in \mathfrak{h} + \mathfrak{g}(l) \} = \{0\}.\end{align*} $$

Hence, we obtain the following result.

Proposition 7.2.1 We have

$$ \begin{align*}(\mathcal{H}^{-\infty})^{H,\chi}= (\mathcal{H}^{-\infty})_Z^{H,\chi}= \mathbb C \delta_0,\end{align*} $$

where $\delta _0 : \mathcal {C}_c^\infty (\mathfrak {r}) \ni \phi \longmapsto \phi (0)$ .

8.1

This section is devoted to the case $ \mathfrak {h}=[ \mathfrak {g}, \mathfrak {g}]$ .

For $\lambda \in F$ , we put $f_\lambda = \lambda e_3^* + e_4^* $ . We have

$$ \begin{align*}\mathfrak{g}( f_\lambda )= \mathrm{Span}\{e_2-\lambda e_3, e_4\}.\end{align*} $$

So that $\mathfrak {h} \in Q(f_\lambda , \mathfrak {g})$ , for all $\lambda \in F$ . We will compute $G.f_\lambda $ . We have

$$ \begin{align*}\exp(-X).Y = x'e_1 + y'e_2 + (z'- xy'+ x'y) e_3+ (t' -xz'+ x'z + \frac{1}{2} x^2 y' -\frac{1}{2} xyx' )e_4.\end{align*} $$

So that

$$ \begin{align*} \exp(X).f_\lambda &= f_\lambda + (\lambda y + z -\frac{1}{2} xy)e_1^* + (-\lambda x +\frac{1}{2} x^2)e_2^* - xe_3^*. \end{align*} $$

Hence,

$$ \begin{align*}G.f_\lambda = \{ f_\lambda + \alpha e_1^* + \beta e_2^* + \gamma e_3^* \, \vert \, \alpha, \beta, \gamma \in F, \, \beta = \lambda \gamma + \frac{1}{2} \gamma^2 \} .\end{align*} $$

This gives that

$$ \begin{align*}(f_\lambda + \mathfrak{h}^\perp) \cap G.f_\lambda = (f_\lambda + \mathfrak{b}^\perp) \cap G.f_\lambda = f_\lambda + F e_1^* = H. f_\lambda.\end{align*} $$

8.2

Finally, we deal with $\mathfrak {h} =\mathrm{Span}\{e_2, e_4\}$ and $f = e_4^*$ .

As $[\mathfrak {h}, \mathfrak {h}]= 0,$ then $\mathfrak {h}$ is subordinate to f and $ f + \mathfrak {h}^\perp = \{ \mu e_1^* + \nu e_3^* + e_4^* \, \vert \, \mu , \nu \in F\}$ .

For $ \mu , \nu \in F$ , we have

$$ \begin{align*}\mathfrak{g}( \mu e_1^* + \nu e_3^* + e_4^* )= \mathrm{Span}\{e_2-\nu e_3, e_4\}.\end{align*} $$

So that $\mathfrak {h} \in Q( \mu e_1^* + \nu e_3^* + e_4^* , \mathfrak {g})$ if and only if $\nu \in F\setminus \{0\}$ . Thus,

$$ \begin{align*}\Xi = \{ \mu e_1^* + \nu e_3^* + e_4^* \, \vert \,\mu \in F, \nu \in F\setminus\{0\} \} .\end{align*} $$

For $\nu \in F$ , we have $ G.f_\nu = \{ f_\nu + \alpha e_1^* + \beta e_2^* + \gamma e_3^* \, \vert \, \alpha , \beta , \gamma \in F, \, \beta = \nu \gamma + \frac {1}{2} \gamma ^2 \}. $

Hereafter, take $\nu \in F\setminus \{0\}$ . We have

$$ \begin{align*} (f + \mathfrak{h}^\perp)\cap G.f_\nu & = \{ f_\nu + \alpha e_1^* + \gamma e_3^* \, \vert \, \alpha, \gamma\in F, \, \nu\gamma + \frac{1}{2} \gamma^2 =0 \} \\ &= \{f_\nu + \alpha e_1^* \, \vert \, \alpha\in F \} \cup \{f_\nu + \alpha e_1^* - 2\nu e_3^* \, \vert \, \alpha\in F \} \\ &= H.f_\nu \cup H.(f_\nu- 2\nu e_3^*). \end{align*} $$

Note that $ f_\nu - 2\nu e_3^* = \exp (2\nu e_1) .f_\nu $ and $ \Xi \supset ( f + \mathfrak {h}^\perp )\cap G.f_\nu $ , although the subalgebra $\mathfrak {h}$ is not a polarization at f nor an ideal of $ \mathfrak {g}$ .

8.3

Put $ \mathfrak {b} = \mathrm{Span}\{e_2, e_3, e_4\},$ which is a polarization at $f_\nu $ . Let $\chi _{f_\nu }$ be the character of $B= \exp (\mathfrak {b})$ defined by

$$ \begin{align*}\chi_{f_\nu}(X)= \psi(\langle f_\nu, X \rangle), \, X \in \mathfrak{b}.\end{align*} $$

Let $\pi _\nu = \mbox {Ind}_B^G\chi _{f_\nu } $ . This representation can be realized in $\mathcal {H} = L^2(F)$ , where the action of G is given by: for $\phi \in L^2(F)$ ,

$$ \begin{align*} & \pi_\nu(\exp(xe_1))\phi(y)= \phi(-x+y), \, x, y \in F\\ & \pi_\nu(\exp(xe_2))\phi(y)=\psi(-\nu xy + \frac{1}{2}xy^2)\phi(y), \, x, y \in F \\ & \pi_\nu(\exp(xe_3))\phi(y)= \psi(\nu x - xy)\phi(y), \, x, y \in F\\ & \pi_\nu(\exp(xe_4))\phi(y)= \psi(x)\phi(y), \, x, y \in F. \end{align*} $$

Let $a \in (\mathcal {H}_{\pi }^{-\infty })^{H,\chi _{f}}$ , $\phi \in \mathcal {C}_c^{\infty }(F),$ and $ h = \exp (xe_2), \, x\in F$ . For each $ y\in F$ , we have $ \pi _\nu (h^{-1}) \phi (y ) = \psi (-\nu xy + \frac {1}{2}xy^2)\phi (y) $ . So that

$$ \begin{align*}\langle \pi_{\nu, -\infty}(h) a , \phi \rangle =\langle a , \pi_\nu(h^{-1})\phi \rangle = \langle a, (y\longmapsto \psi(-\nu xy + \frac{1}{2}xy^2)\phi(y) )\rangle .\end{align*} $$

On the other hand, a is $\chi _{f}$ -semi-invariant:

$$ \begin{align*}\langle \pi_{\nu, -\infty}(h) a , \psi \rangle = \langle \chi_{f}(h^{-1}) a , \psi \rangle = \langle a , \psi \rangle. \end{align*} $$

Hence, a satisfies the following condition:

$$ \begin{align*} \langle a, (y\longmapsto (\psi(-\nu xy + \frac{1}{2}xy^2)-1)\phi(y)) \rangle = 0 , \quad \forall \phi \in \mathcal{C}_c^{\infty}(F), x \in F. \end{align*} $$

We deduce that $\mbox {Supp}(a) \subset \{0, 2\nu \}$ . Thus, we obtain the following proposition.

Proposition 8.3.1 We have

$$ \begin{align*}(\mathcal{H}^{-\infty})^{H,\chi}= (\mathcal{H}^{-\infty})_Z^{H,\chi}= \mathbb C a_0 + \mathbb C a_{2\nu},\end{align*} $$

where $a_0 : \mathcal {C}_c^\infty (F) \ni \phi \longmapsto \phi (0)$ and $ a_{2\nu } : \mathcal {C}_c^\infty (F) \ni \phi \longmapsto \phi (2\nu )$ .