We provide a microlocal necessary condition for distinction of admissible representations of real reductive groups in the context of spherical pairs.
Let
${\mathbf {G}}$ be a complex algebraic reductive group and
${\mathbf {H}}\subset {\mathbf {G}}$ be a spherical algebraic subgroup. Let
${\mathfrak {g}},{\mathfrak {h}}$ denote the Lie algebras of
${\mathbf {G}}$ and
${\mathbf {H}}$, and let
${\mathfrak {h}}^{\bot }$ denote the orthogonal complement to
${\mathfrak {h}}$ in
${\mathfrak {g}}^*$. A
${\mathfrak {g}}$-module is called
${\mathfrak {h}}$-distinguished if it admits a nonzero
${\mathfrak {h}}$-invariant functional. We show that the maximal
${\mathbf {G}}$-orbit in the annihilator variety of any irreducible
${\mathfrak {h}}$-distinguished
${\mathfrak {g}}$-module intersects
${\mathfrak {h}}^{\bot }$. This generalises a result of Vogan [Vog91].
We apply this to Casselman–Wallach representations of real reductive groups to obtain information on branching problems, translation functors and Jacquet modules. Further, we prove in many cases that – as suggested by [Pra19, Question 1] – when H is a symmetric subgroup of a real reductive group G, the existence of a tempered H-distinguished representation of G implies the existence of a generic H-distinguished representation of G.
Many of the models studied in the theory of automorphic forms involve an additive character on the unipotent radical of the subgroup
$\bf H$, and we have devised a twisted version of our theorem that yields necessary conditions for the existence of those mixed models. Our method of proof here is inspired by the theory of modules over W-algebras. As an application of our theorem we derive necessary conditions for the existence of Rankin–Selberg, Bessel, Klyachko and Shalika models. Our results are compatible with the recent Gan–Gross–Prasad conjectures for nongeneric representations [GGP20].
Finally, we provide more general results that ease the sphericity assumption on the subgroups, and apply them to local theta correspondence in type II and to degenerate Whittaker models.