1.1 Proof strategy

We now give an overview of the proof of TheoremA and TheoremB. The proof can be organized into five main steps:

1. (0) description of the combinatorial setup: $\mathcal {C}, \mathcal {T},\mathcal {E}$ ;

2. (1) orientation on $\mathcal {E}$ , fork lemma and connectivity of the regions $\Sigma$ ;

3. (2) growth of traces for neighbors around a region and for escaping rays, and definition of an attracting subgraph of $\mathcal {E}$ ;

4. (3) Fibonacci growth and equivalent characterizations of Bowditch representations;

5. (4) proof that the set $\mathfrak {X}_{BQ}$ strictly contains convex cocompact representations.

1. Step (0): We start with a discussion of the combinatorial setup needed for our study. Given the one-hole torus $S:=S_{1,1}$ , we define three simplicial complexes associated to $S$ : the curve complex $\mathcal {C}$ , its dual $\mathcal {T}$ and the edge graph $\mathcal {E}$ . The curve complex $\mathcal {C}$ is the two-dimensional simplicial complex, where $k$ -simplices are given by $k+1$ distinct (free homotopy classes of) simple closed curves in $S$ that pairwise intersect once. Its simplicial dual $\mathcal {T}$ is a countably infinite tri-valent simplicial tree properly embedded in $\mathbf {H}^2$ defined by the property that the set $\mathcal {T}^{(k)}$ of $k$ -simplices in $\mathcal {T}$ satisfies $\mathcal {T}^{(k)} = \mathcal {C}^{(2-k)}$ for $k=0,1,2$ . Finally, the edge graph $\mathcal {E}$ has vertices corresponding to edges of $\mathcal {T}$ , and edges corresponding to adjacency, so one can see that $\mathcal {E}$ is a ‘trivalent tree of triangles’ in the sense that it is a graph made of triangles (one for each triangle $T$ in $\mathcal {C}^{(2)}$ or, equivalently, one for each vertex of $\mathcal T$ ). See Figure 1.

2. Step (1): For the first step, we show how any representation in $\mathfrak {X}_c$ induces an orientation on the edges of $\mathcal {E}$ as follows. An edge $(X, Y, Z; T, T) \in \mathcal {E}$ indicates that the edge is in the triangle $(X, Y, Z) \in \mathcal {C}^2$ and is between the triangles $(X, Y, T), (X, Z, T') \in \mathcal {C}^2$ which are adjacent to $(X, Y, Z)$ , see Figure 2. If $t = |\mathrm {tr} (\rho (T)) | \gt |\mathrm {tr} (\rho (T'))| = t'$ , then the oriented edge points towards $T'$ . In the case where there is an equality, one can choose the orientation arbitrarily (and the choice will not affect the result).

   We have a $4$ -coloring of the edges of $\mathcal {E}$ such that at each vertex there are only two triangles of two different colors meeting. We say that a vertex $v \in \mathcal {E}^{(0)}$ is a fork if there are (at least) two arrows of different colors pointing away from $v$ . The Fork Lemma (Lemma3.4) shows that if a vertex $v = (X, Y) \in \mathcal {E}^{(0)}$ is a fork, then $\min \{|x|, |y| \} \leqslant \max \{6,\sqrt {|\Re (c)|}\}$ for $x=\textrm {tr }(\rho (X))$ and $y=\textrm {tr }(\rho (Y))$ . We use this result to show, for $[\rho ] \in \mathfrak {X}_c$ , the set $\Omega _\rho (K)$ of regions with traces smaller than $K \geqslant \max \{6,\sqrt {|\Re (c)|}\}$ is connected. The proof is by contradiction and considers the distance between two connected components. The Fork Lemma is used to reach the conclusion when such a distance is greater than $2$ since in that case a fork will appear.

3. Step (2): For the second step we first study the behavior of the values of the (norm of the) traces of regions that are all neighbors of a central region $X$ . Given $[\rho ] \in \mathfrak {X}_c$ we see that if we denote by $(X_n)_{n \in \mathbb {Z}}$ the regions around $X$ , then the behavior of the sequence $(u_n)_{n\in \mathbb {N}}$ defined by $u_n = \mathrm {tr} (\rho (X_n))$ depends on the value of $f(x)$ where $x=\textrm {tr }(\rho (X))$ and $f(x) = \mathrm {Res} (\chi _x, \chi _x')$ . In fact if $f(x)=0$ , then $(u_n)_{n\in \mathbb {N}}$ grows at most quadratically, while if $f(x) \lt 0$ , then $(u_n)_{n\in \mathbb {N}}$ remains bounded. Finally, in Lemma3.8 we show that there exists a constant $D = D(c)$ , such that if the sequence $(u_n)_{n\in \mathbb {N}}$ defined above with $f(x) \gt 0$ does not grow exponentially in both directions, then there are infinitely many terms of the sequence such that $|u_n| \lt D$ . We then consider escaping rays, that is infinite geodesic rays where each edge $e_n$ is directed from $v_n$ to $v_{n+1}$ . In Lemma3.11 we show that if $\{ e_n \}_{n\in \mathbb {N}}$ is an escaping ray, then there are two cases:

   1. (1) there exists a region $\alpha$ such that the ray is eventually contained in the boundary of $\alpha$ , such that $f(x) \leqslant 0$ with $x = \mathrm {tr} (\rho (\alpha ))$ ; or

   2. (2) the ray meets infinitely many distinct regions with trace smaller than $M(c)$ , where $M(c):=\max \{6, \sqrt {|\Re (c)|}, D(c)\}$ .

   Using the description of the growth of the traces for regions around a fixed central trace, we define, for every representation $[\rho ] \in \mathfrak {X}_c$ and for every $K\gt M(c)$ , a connected attracting subgraph $T_\rho (K)$ of $\mathcal {E}$ . (Recall that $\mathcal {E}$ is a ‘trivalent tree of triangles’, so we want to define $T_\rho (K)$ so that it is a union of ‘triangles’.) The connectivity of $T_\rho (K)$ comes from the connectivity of $\Omega _\rho (K)$ , while the fact that $T_\rho (K)$ is attractive comes from the definition of $T_\rho (K)$ . We then show that for representations in $\mathfrak {X}_{BQ}$ , that is representations satisfying conditions (BQ1) and (BQ2), such a graph $T_\rho (K)$ is finite for all $K\gt M(c)$ .

4. Step (3): In the penultimate step, given a vertex $v \in \mathcal {E}^{(0)}$ , we define a Fibonacci function $F_v : \Omega \rightarrow \mathbb {N}$ . We then show that this function has the following property: if $\{\alpha, \beta \}$ is a set of free generators for $F_2$ corresponding to regions $X$ and $Y$ such that $v =(X, Y) \in \mathcal {E}^{(0)}$ , then for any element $\gamma \in \Omega$ the Fibonacci function corresponds to the word length of $\gamma$ with respect to $\{\alpha, \beta \}$ , that is $F_v(\gamma ) = W(\gamma )$ . Note that we only care about the asymptotic growth of this function, as you will see, so the choice of $v$ will not affect our results. We then define what it means for a function $g \colon \thinspace \Omega \rightarrow [0, \infty )$ to have Fibonacci growth, i.e. there exist constants $\kappa _1, \kappa _2 \gt 0$ such that

   \begin{equation*}\kappa _1 F_v (X) \leqslant g(X) \leqslant \kappa _2 F_v (X)\end{equation*}
   for all $X \in \Omega$ . If only the lower (respectively, upper) bound is satisfied, the function $g$ is said to have a lower (respectively, upper) Fibonacci growth. Given any $[\rho ] \in \mathfrak {X}$ , we denote by $\phi _\rho \colon \thinspace \Omega \rightarrow \mathbb {C}$ the function $\phi _\rho (X) := \log | \mathrm {tr} (\rho (X)) |$ . We consider the function $\phi _\rho ^+ := \max \{ \phi _\rho, 0 \}$ . We then show that this function always has an upper Fibonacci growth, but for representations in $[\rho ] \in \mathfrak {X}_{BQ}$ we show that the function has upper and lower Fibonacci growth.

   We use this result to show that we can characterize representations in $\mathfrak {X}_{BQ}$ in terms of the attracting graph $T_\rho (K)$ as follows: $[\rho ]$ is in $(\mathfrak {X}_c)_{BQ}$ if and only if the subgraph $T_\rho (K)$ is finite for all $K\gt M(c)$ , which proves TheoremB. This result is crucial to show that the set $\mathfrak {X}_{BQ}$ is open and the mapping class group acts on it properly discontinuously, and hence to prove TheoremA.

5. Step (4): In order to show that the set $\mathfrak {X}_{BQ}$ strictly contains convex cocompact representations, we use representations described by Will [Reference WillWil07, Reference WillWil12]: a family of discrete, faithful and type-preserving representations of the once-punctured torus in $\mathfrak {X}(F_2, \mathrm {SU}(2,1))$ . These representations will take non-peripheral, non-trivial simple closed curves to loxodromic elements and will map the commutator to a parabolic unipotent element. In order to see that these representations are in $\mathfrak {X}_{BQ}$ , we use a characterization of both sets in term of the growth of $|l (\rho (\gamma ))|$ : the existence of $k, m \gt 0$ such that $|l (\rho (\gamma ))| \geqslant k W(\gamma ) - m$ for all $\gamma \in F_2$ corresponding to non-peripheral, non-trivial simple closed curves in $S_{1,1}$ . On the other hand, since the (peripheral) simple closed curve corresponding to the commutator is mapped to a unipotent element, we can see these representations are not convex cocompact. There are also other examples of representations that are in $\mathfrak {X}_{BQ} \setminus \mathfrak {X}_{CC}$ and such that the peripheral element is mapped to a parabolic non-unipotent element, see Falbel and Parker [Reference Falbel and ParkerFP03], Falbel and Koseleff [Reference Falbel and KoseleffFK02b, Reference Falbel and KoseleffFK02a] or Parker and Gusevskii [Reference Gusevskii and ParkerGP03, Reference Gusevskii and ParkerGP00]. See Will [Reference WillWil16] for a discussion of these references.

Figure 1.

The complexes $\mathcal {C}$ (in gray) and $\mathcal {T}$ (in black), and the labeling of the connected components of $\mathbf {H}^2 \setminus \mathcal {T}$ once we fix the generators of $F_2 = \langle \alpha, \beta \rangle$ .

Figure 2.

Oriented edge $(X, Y, Z; T \longrightarrow T')$ . If you forget the orientation, the same red edge can be denoted $(X; Y, Z)$ .

1.2 General setting

Let $\Gamma$ be a finitely presentable group with the discrete topology and let $G$ be a Lie group. The set $\mathrm {Hom}(\Gamma, G)$ of homomorphisms from $\Gamma$ to $G$ is a topological space by giving it the compact-open topology. The group $G$ acts on $\mathrm {Hom}(\Gamma, G)$ by conjugation, that is, $g\cdot \rho := \iota _g\circ \rho$ where $\iota _g\in \mathrm {Inn}(G)$ . Let $\mathrm {Orb}_G(\rho )$ be the $G$ -orbit of $\rho$ . Define the subspace of polystable homomorphisms

\begin{equation*}\mathrm {Hom}(\Gamma, G)^*:=\{\rho \in \mathrm {Hom}(\Gamma, G)\ |\ \mathrm {Orb}_G(\rho )=\overline {\mathrm {Orb}_G(\rho )}\},\end{equation*}

and define the polystable quotient space

\begin{equation*}\mathfrak {X}(\Gamma, G):=\mathrm {Hom}(\Gamma, G)^*/G.\end{equation*}

This quotient is called the $G$ -character variety of $\Gamma$ for historical reasons, even though in this generality it may be neither a variety, see Casimiro et al. [Reference Casimiro, Florentino, Lawton and OliveiraCFLO16], nor correspond to characters, see Lawton and Sikora [Reference Lawton and SikoraLS17].

Let $\mathcal {P}$ be a collection of properties such that if $\rho \in \mathrm {Hom}(\Gamma, G)$ satisfies $\mathcal {P}$ , then $\iota _g\circ \rho$ also satisfies $\mathcal {P}$ for all $g\in G$ . Then $G$ acts on the subspace

\begin{equation*}\mathrm {Hom}_{\mathcal {P}}(\Gamma, G):=\{\rho \in \mathrm {Hom}(\Gamma, G)\ |\ \rho \ \mathrm { satisfies }\ \mathcal {P}\}\end{equation*}

by conjugation. The polystable quotient of $\mathrm {Hom}_{\mathcal {P}}(\Gamma, G)$ by $G$ , denoted by $\mathfrak {X}_{\mathcal {P}}(\Gamma, G)$ , is called the $\mathcal {P}$ -relative $G$ -character variety of $\Gamma$ . If $\mathcal {P}$ is vacuous, one gets back $\mathfrak {X}(\Gamma, G).$

Let $\mathrm {Aut}_{\mathcal {P}}(\Gamma )$ denote the subgroup of $\mathrm {Aut}(\Gamma )$ that preserves $\mathrm {Hom}_{\mathcal {P}}(\Gamma, G)$ . Then $\mathrm {Inn}(\Gamma )\subset \mathrm {Aut}_{\mathcal {P}}(\Gamma )$ and so $\mathrm {Out}_{\mathcal {P}}(\Gamma ):=\mathrm {Aut}_{\mathcal {P}}(\Gamma )/\mathrm {Inn}(\Gamma )$ acts on $\mathfrak {X}_{\mathcal {P}}(\Gamma, G)$ by $[\alpha ]\cdot [\rho ]=[\rho \circ \alpha ^{-1}]$ . Again, if $\mathcal {P}=\emptyset$ , one obtains an action of $\mathrm {Out}(\Gamma )$ on $\mathfrak {X}(\Gamma, G)$ .

One of the main theorems in Rapinchuk [Reference RapinchukRap15] says that, for any complex affine variety $V$ , there exists a finitely generated group $\Gamma$ and a complex reductive group $G$ , so that $\mathfrak {X}(\Gamma, G)-\{\chi _0\}$ and $V$ are isomorphic where $\chi _0$ is the point in the character variety corresponding to the trivial representation. Let $\mathcal {F}$ be the collection of finitely generated groups and $\mathcal {G}$ the collection of complex reductive affine algebraic groups. Then, since the $G$ -character variety of the trivial group $\Gamma =\{e\}$ is a point for any $G$ , the set $\{\mathfrak {X}(\Gamma, G)\ |\ \Gamma \in \mathcal {F},\ G\in \mathcal {G}\}$ generates the Grothendieck ring of affine varieties. By Bumagin and Wise [Reference Bumagin and WiseBW05] all countable groups arise as $\mathrm {Out}(\Gamma )$ as we vary finitely generated $\Gamma$ . So we have a very rich class of natural dynamical systems $(\mathfrak {X}(\Gamma, G),\mathrm {Out}(\Gamma ))$ which are interesting only when both $\mathfrak {X}(\Gamma, G)$ and $\mathrm {Out}(\Gamma )$ are non-trivial.

1.3 Some history

A classical example to consider is when $G=\mathrm {U}(1)$ and $\Gamma =\Gamma _g$ is the fundamental group of a closed orientable surface $S_g$ of genus $g$ . Then $\mathfrak {X}(\Gamma _g, G)=\mathrm {Hom}(\Gamma _g, \mathrm {U}(1))\cong \mathrm {U}(1)^{2g}$ which is identified with the Jacobian of $S_g$ , parametrizing topologically trivial holomorphic line bundles over a Riemannian surface diffeomorphic to $S_g$ . The action of $\mathrm {Out}(\Gamma _g)$ factors through $\mathrm {GL}(2g,\mathbb {Z})$ acting on a product of circles, and is classically ergodic. The Dehn–Nielsen–Baer Theorem [Reference Farb and MargalitFM11] states that for such surfaces the (extended) mapping class group of isotopy classes of all homeomorphisms of $S_g$ is isomorphic to $\mathrm {Out}(\Gamma _g)$ .

The above example generalizes to arbitrary compact Lie groups $K$ as follows. The smooth locus of the character variety $\mathfrak {X}(\Gamma _g,K)$ is symplectic, as proven by Goldman [Reference GoldmanGol84], and the associated invariant measure is finite, as shown by Huebschmann [Reference HuebschmannHue95]. With respect to this measure, $\mathrm {Out}(\Gamma _g)$ acts ergodically on $\mathfrak {X}(\Gamma _g,K)$ when $g\geqslant 2$ , see Goldman [Reference GoldmanGol97] and Pickrell and Xia [Reference Pickrell and XiaPX02]. For non-orientable surfaces $N$ with $\chi (N)\leqslant -2$ and $G=\mathrm {SU}(2)$ , Palesi [Reference PalesiPal11] shows ergodicity with respect to a different invariant measure.

The situation for non-compact Lie groups $G$ is even more complicated, and we need two definitions to describe the situation. First, for a semisimple Lie group $G$ with no compact factors and a rank $r$ free group $F_r$ , a homomorphism $\rho : F_r \longrightarrow G$ is said to be primitive stable if any bi-infinite geodesic in the Cayley graph of $F_r$ defined by a primitive element (a member of free generating set) is mapped under the orbit map to a uniformly Morse quasigeodesic in the symmetric space $G/K$ where $K$ is a maximal compact in $G$ . Second, if $\rho$ is injective, $\rho (F_r)$ is discrete, and $\rho (F_r)$ preserves and acts cocompactly on some non-empty convex subset of $G/K$ then $\rho$ is convex cocompact.

In the case of $G=\mathrm {SL}(2,\mathbb {C})$ , Minsky [Reference MinskyMin13] showed the set $\mathfrak {X}_{PS}(F_{n},G)$ of primitive stable representations is a domain of discontinuity for the action of $\mathrm {Out}(F_r)$ on $\mathfrak {X}(F_{r},G)$ . Minsky also proved that $\mathfrak {X}_{PS}(F_{n},G)$ strictly contains the set $\mathfrak {X}_{CC}(F_{r},G)$ of convex cocompact representations that is also a domain of discontinuity for the action of $\mathrm {Out}(F_r)$ on $\mathfrak {X}(F_{r},G)$ . When $\Gamma = F_2$ is the fundamental group of the one $1$ -holed torus $S_{1,1}$ , Bowditch [Reference BowditchBow98], in the case of quasi-Fuchsian representations, and more generally Tan, Wong and Zhang [Reference Tan, Wong and ZhangTWZ08], showed that there is a domain of discontinuity $\mathfrak {X}_{BQ}(F_{2},G)$ for the action of the mapping class group of $S_{1,1,}$ on the corresponding character variety $\mathfrak {X}(F_{2},G)$ . The set $\mathfrak {X}_{BQ}(F_{2},G)$ is the set of Bowditch representations where the comparable conditions (for $G=\mathrm {SL}(2,\mathbb {C})$ ) from TheoremB hold true (although [Reference BowditchBow98, Reference Tan, Wong and ZhangTWZ08] do not show $\mathfrak {X}_{BQ}(F_{2},G)$ strictly contains the convex cocompact representations). Later, Lee and Xu [Reference Lee and XuLX20] and Series [Reference SeriesSer19, Reference SeriesSer20] proved $\mathfrak {X}_{PS}(F_{2},G)=\mathfrak {X}_{BQ}(F_{2},G)$ . Schlich [Reference SchlichSch22] generalized this result to cases where $G$ is the group of isometries of a $\delta$ -hyperbolic space. We will discuss her results below in more detail, since they are connected with the results we present.

Following the work of Bowditch, there have been other cases where interesting domains of discontinuity have been described for relative $\mathrm {SL}(2,\mathbb {C})$ -character varieties of surface groups. For example, you can see the description of a domain of discontinuity, the Bowditch set $\mathfrak {X}_{BQ}(\pi _1(S), G)$ , for the action of the mapping class group $\mathrm {Mod}(S)$ on the variety $\mathfrak {X}(\pi _1(S), G)$ , by Maloni, Palesi and Tan [Reference Maloni, Palesi and TanMPT15] when $S$ is four $4$ -holed sphere and by Maloni and Palesi [Reference Maloni and PalesiMP20] when $S$ is the three $3$ -holed projective plane. In the case when $\Gamma$ is the fundamental group of a hyperbolic $3$ -manifold there has also been significant work finding interesting domains of discontinuity as described in this survey of Canary [Reference CanaryCan13]. When $G$ is a semisimple group of higher rank and without compact factors, Kim and Kim [Reference Kim and KimKK20] generalized Minsky’s work on primitive-stable representations and showed that they form an open domain of discontinuity for the action of $\mathrm {Out}(\Gamma )$ on $\mathfrak {X}(\Gamma, G)$ when $\Gamma$ is a free group of rank at least $2$ or the fundamental group of a closed surface of genus $g\geqslant 2$ .

Motivated by Lubotzky [Reference LubotzkyLub11], Minsky [Reference MinskyMin13] and Gelander and Minsky [Reference Gelander and MinskyGM13] discussed primitive-stable and redundant representations in the case of free groups and $G=\mathrm {SL}(2,\mathbb {C})$ . Inspired by their work, one can imagine a decomposition of $\mathfrak {X}(\Gamma, G)$ into an open set where the action of $\mathrm {Out}(\Gamma )$ is properly discontinuous and a closed set where the action is chaotic. The only example fully understood where $G$ is not compact that exemplifies such a dynamical dichotomy is when $\Gamma$ is a free group of rank 2 and $G = \mathrm {SL}(2,\mathbb {R})$ , see Goldman [Reference GoldmanGol03], Goldman et al. [Reference Goldman, McShane, Stantchev and TanGMST19] and Marché and Wolff [Reference Marché and WolffMW16, Reference Marché and WolffMW19].

As summarized in Goldman [Reference GoldmanGol06], when $G$ is compact, (relative) character varieties generally exhibit non-trivial homotopy, and the action of the mapping class group exhibits non-trivial chaotic dynamics. In contrast, when $G$ is non-compact, then $\mathfrak {X}(\Gamma _g,G)$ contains open sets (like the Teichmüller space or more generally Hitchin components) which are contractible and admit properly discontinuous actions. Philosophically, the action of $\mathrm {Out}(\Gamma )$ on (relative) character varieties exhibits properties of these two extremes.

1.4 Relation with work of Schlich

Schlich [Reference SchlichSch22] considers representations from the fundamental group $\Gamma = \pi _1(S)$ of the one $1$ -holed torus $S=S_{1,1}$ into the isometry group $G$ of a geodesic and proper $\delta$ -hyperbolic space. Since ${\rm PSU}(2,1) =\mathrm {Isom}(\mathbf {H}^2_{\mathbb {C}})$ is the isometry group of the complex hyperbolic plane (see Section 2.3), her result applies to the setting we study. For the character variety $\mathfrak {X} = \mathfrak {X}(F_2,\mathrm {SU}(2,1))$ , Schlich defines two subsets of the character variety: the Bowditch set (using Definition (BQ3)) and the set of primitive-stable representations (generalizing Minsky’s definition to this setting). One of the main results of her thesis and paper shows that the two sets coincide. Since it is not hard to see that the set of primitive-stable representations is a domain of discontinuity for the action of $\mathrm {Out}(F_2)$ on $\mathfrak {X}$ , she also proves, as a corollary of her result, that the set of Bowditch representations also has that property. In her thesis, she also considers the case of $S = S_{0,4}$ , a four-holed sphere, and proves that in that setting the set of ‘simple-stable’ representations coincides with the set of Bowditch’s representations [Reference SchlichSch23].

The main difference between our work and Schlich’s work is in the way Bowditch representations are defined. This also differentiates the challenging aspects of the work. We define our Bowditch set explicitly in terms of the character variety using the symmetry and structure understood from [Reference LawtonLaw06, Reference LawtonLaw07]. Our approach is similar to that of [Reference BowditchBow98] and its generalizations [Reference Tan, Wong and ZhangTWZ08, Reference Maloni, Palesi and TanMPT15, Reference Maloni and PalesiMP20], but adapted to representations into $\mathrm {SU}(2,1)$ where the character variety is significantly more complicated. In particular, our definition is well-adapted to computer implementations and testing since it is a definition dependent on only a finite number of conditions. Schlich’s work generalizes to isometry groups of Gromov hyperbolic spaces the results obtained for $\mathrm {SL}(2,\mathbb {C})$ by Series [Reference SeriesSer19, Reference SeriesSer20]and Lee and Xu [Reference Lee and XuLX20], that for the free group $F_2$ the notions of Bowditch representations and primitive stable representations are equal. In fact, her result gives a new independent proof of this result for $\mathrm {SL}(2,\mathbb {C})$ .