2.1 Preliminaries

We begin by stating the correspondence between (absolutely simple) abelian varieties with real multiplication over ${\mathbf {Q}}$ and weight- $2$ newforms for $\Gamma _0(N)$ .

Recall that $L(A/{\mathbf {Q}}, s)$ denotes the L-series of A and $L(f, s)$ denotes the L-series of f and that $L(A/{\mathbf {Q}},s)$ is defined as

$$ \begin{align*} L(A/{\mathbf{Q}},s) &= \prod_p\frac{1}{\det(1 - \operatorname{Frob}_p^{-1} p^{-s} \mid {\operatorname{H}}_{\mathrm{\acute{e}t}}^1(A \otimes {\overline{{\mathbf{Q}}}}, {\mathbf{Q}}_\ell)^{I_p})}, \end{align*} $$

where for each Euler factor at p, one chooses a prime $\ell \neq p$ for the $\ell $ -adic cohomology group; this is well-defined because the Euler factors are independent of $\ell $ . The product converges for $\operatorname {Re}(s)> \frac {3}{2}$ to a holomorphic function. The L-function associated to $f = \sum _n a_n q^n$ with coefficients $a_n \in {\mathbf {C}}$ is

$$ \begin{align*} L(f,s) &= \sum_{n \ge 1}\frac{a_n}{n^s} = \prod_p\frac{1}{1 - a_pp^{-s} + \varepsilon(p)p^{1-2s}}, \end{align*} $$

where $\varepsilon (p)$ is $1$ if $p \nmid N$ and $0$ otherwise. Since $L(f,s)$ is the Mellin transform of f and f is a cusp form, $L(f, s)$ is holomorphic on the whole complex plane.

We will use these equivalences tacitly. Note that in the literature, sometimes more general modular abelian varieties are considered, which are quotients of $J_1(N)$ and which can have complex multiplication.

However, when A is an absolutely simple abelian surface with CM, then A cannot be of $\mathrm {GL}_2$ -type over ${\mathbf {Q}}$ , as the following result shows. We thank Pip Goodman for pointing it out to us.

Proof. Let $E = \mathrm {End}_{{\overline {{\mathbf {Q}}}}}^0(A) = \mathrm {End}_{{\overline {{\mathbf {Q}}}}}(A) \otimes _{\mathbf {Z}} {\mathbf {Q}}$ be the geometric endomorphism algebra of A. By [Reference Shimura101, Proposition 30], the minimal field over which the endomorphisms of A are defined is $E^*$ , the reflex field of E (note that the base field is just ${\mathbf {Q}}$ here). In particular, $E^*|{\mathbf {Q}}$ is Galois, and the absolute Galois group ${ {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}|{\mathbf {Q}})}$ acts on $\mathrm {End}_{{\overline {{\mathbf {Q}}}}}^0(A)$ through $ {\mathrm {Gal}}(E^*|{\mathbf {Q}})$ , so we obtain an embedding

$$\begin{align*}i \colon \operatorname{\mathrm{Gal}}(E^*|{\mathbf{Q}}) \hookrightarrow \operatorname{\mathrm{Aut}}(\mathrm{End}_{{\overline{{\mathbf{Q}}}}}^0(A)) \,. \end{align*}$$

We now consult Examples 8.4 (2) in loc. cit. In Example (C), there the reflex field is not Galois, and in Example (A), the CM-type is not primitive, which means that A is not absolutely simple. So both these cases cannot occur, and by Example (B), it follows that $E^* = E$ ; in particular, the map i above is an isomorphism. This finally implies that

$$\begin{align*}\mathrm{End}^0_{\mathbf{Q}}(A) = \mathrm{End}_{{\overline{{\mathbf{Q}}}}}^0(A)^{\operatorname{\mathrm{Gal}}(E^*/{\mathbf{Q}})} = E^{\operatorname{\mathrm{Gal}}(E/{\mathbf{Q}})} = {\mathbf{Q}} \,. \end{align*}$$

Remark 2.3. In the situation of theorem 2.1, A is isogenous to $A_f$ (by Faltings’ Isogeny Theorem), and therefore, the Galois representations on $V_{\mathfrak {p}} A$ and on $V_{\mathfrak {p}} A_f$ are isomorphic (similarly for $V_p A$ and $V_p A_f$ ). When A and/or f are clear from the context, we write $\rho _{\mathfrak {p}^\infty }$ and $\rho _{p^\infty }$ for these representations, which depend only on the Galois orbit of f. The fact that A and $A_f$ are isogenous also implies that the semi-simplifications of $\rho _{\mathfrak {p},A}$ and of $\rho _{\mathfrak {p},f} := \rho _{\mathfrak {p},A_f}$ are isomorphic when $\mathfrak {p}$ is a regular prime of both ${\mathbf {Z}}[f] \simeq \mathrm {End}_{{\mathbf {Q}}}(A_f)$ and $\mathrm {End}_{{\mathbf {Q}}}(A)$ (and similarly for $\rho _{p,A}$ and $\rho _{p,f}$ ).

Note that the canonical isomorphism ${\mathbf {Z}}[f] \simeq {\mathbf {T}}_{\mathbf {Z}}/I_f \simeq \mathrm {End}_{{\mathbf {Q}}}(A_f)$ induces a canonical identification of ${\mathbf {Q}}(f)$ with $\mathrm {End}^0_{{\mathbf {Q}}}(A_f)$ , which in turn is isomorphic to $\mathrm {End}^0_{{\mathbf {Q}}}(A)$ via the isogeny between A and $A_f$ . Fixing the isogeny, this identifies $\mathcal {O} = \mathrm {End}_{{\mathbf {Q}}}(A)$ with an order in the totally real number field ${\mathbf {Q}}(f)$ . If ${\mathbf {Z}}[f]$ is contained in $\mathcal {O}$ under this identification (e.g., when $\mathcal {O}$ is the maximal order), then the Fourier coefficient $a_n$ of f, which is the image of the Hecke operator $T_n \in {\mathbf {T}}_{{\mathbf {Z}}}$ in ${\mathbf {Z}}[f]$ , can be interpreted as an element of $\mathcal {O}$ (i.e., an endomorphism of A). We make use of this to get the correct identifications of $\sigma $ -isotypic components when dealing with the Gross–Zagier formula for the height of a Heegner point in Section 3.7.

Write $F = {\mathbf {Q}}(f)$ and $\mathcal {O}_F$ for the maximal order of F, and let $\mathcal {O} \subseteq \mathcal {O}_F$ be any order in F. Recall the conductor ideal of $\mathcal {O}$ in $\mathcal {O}_F$ ,

$$\begin{align*}\mathfrak{f}(\mathcal{O}_F/\mathcal{O}) = \{a \in \mathcal{O} : a \mathcal{O}_F \subseteq \mathcal{O}\} \,; \end{align*}$$

it is the largest ideal of $\mathcal {O}$ that is also an ideal of $\mathcal {O}_F$ . If A is an abelian variety such that $\mathrm {End}_{\mathbf {Q}}(A) \cong \mathcal {O}$ , then one can check that the isogenous abelian variety $A' := A/A[\mathfrak {f}(\mathcal {O}_F/\mathcal {O})]$ has $\mathrm {End}_{\mathbf {Q}}(A') \cong \mathcal {O}_F$ . So by working with $A'$ instead of with A (or $A_f$ ), we can assume that the endomorphism ring is the maximal order.

Definition 2.4. Let p be a prime. We write

$$\begin{align*}\chi_{p^n} \colon {\operatorname{\mathrm{Gal}}({\overline{{\mathbf{Q}}}}|{\mathbf{Q}})} \to \operatorname{\mathrm{Aut}}(\mu_{p^n}({\overline{{\mathbf{Q}}}})) \simeq ({\mathbf{Z}}/p^n{\mathbf{Z}})^\times \end{align*}$$

for the mod- $p^n$ cyclotomic character and

$$\begin{align*}\chi_{p^\infty} \colon {\operatorname{\mathrm{Gal}}({\overline{{\mathbf{Q}}}}|{\mathbf{Q}})} \to \operatorname{\mathrm{Aut}}(\mu_{p^\infty}({\overline{{\mathbf{Q}}}})) \simeq {\mathbf{Z}}_p^\times \end{align*}$$

for the p-adic cyclotomic character.

Theorem 2.6 (Characteristic polynomials of Frobenii of a modular Galois representation)

Let $f \in S_2(\Gamma _0(N))$ be a newform with Fourier coefficients $a_\ell $ and coefficient field ${\mathbf {Q}}(f)$ a totally real field of degree g.

Associated to f, there is a strictly compatible system of $\mathfrak {p}$ -adic Galois representations $\rho _{\mathfrak {p}^\infty }$ , unramified outside $N p(\mathfrak {p})$ . For all $\ell \nmid N p(\mathfrak {p})$ , the characteristic polynomial of $\rho _{\mathfrak {p}^\infty }(\operatorname {Frob}_\ell )$ equals

$$\begin{align*}\operatorname{charpol}(f, \ell; T) := \det\bigl(T - \rho_{\mathfrak{p}^\infty}(\operatorname{Frob}_\ell)\bigr) = T^2 - a_\ell T + \ell \in {\mathbf{Z}}[f][T] \,. \end{align*}$$

One has

$$\begin{align*}\det \circ \rho_{\mathfrak{p}^\infty} = \chi_{p^\infty} \end{align*}$$

for $\mathfrak {p} \nmid N$ . In particular, $\rho _{\mathfrak {p}^\infty }$ is odd. The determinant of the p-adic Galois representation

$$\begin{align*}\rho_{p^\infty} \colon {\operatorname{\mathrm{Gal}}({\overline{{\mathbf{Q}}}}|{\mathbf{Q}})} \to \mathrm{GL}_{2g}({\mathbf{Z}}_p) \end{align*}$$

is $\chi _{p^\infty }^g$ .

If $\mathfrak {p}$ is a regular prime of ${\mathbf {Z}}[f]$ (or $\mathcal {O}$ ) not dividing $N\ell $ , then we write $\operatorname {charpol}(f, \ell , \mathfrak {p}; T)$ for the characteristic polynomial of the image of $\operatorname {Frob}_\ell $ under $\rho _{\mathfrak {p},f}$ (or $\rho _{\mathfrak {p},A}$ ); it is the image of $\operatorname {charpol}(f, \ell; T)$ in ${\mathbf {F}}_{\mathfrak {p}}[T]$ .

Magma can compute the Fourier coefficients $a_\ell $ of a newform f and its coefficient ring ${\mathbf {Z}}[f]$ efficiently. This will be crucial for computing the image of $\rho _{\mathfrak {p}}$ , because the only access to elements of the absolute Galois group of ${\mathbf {Q}}$ we have is via Frobenius elements, and we can reconstruct the characteristic polynomials of the Frobenii acting on $A[\mathfrak {p}]({\mathbf {Q}})$ , which uniquely determine their semi-simple part. The fact that we know only the characteristic polynomials also means that we do not have direct access to the unipotent part of $\rho _p(\operatorname {Frob}_\ell )$ via $a_\ell $ alone.

Let p be a prime. We fix an embedding of ${\overline {{\mathbf {Q}}}}$ into ${\overline {{\mathbf {Q}}}}_p$ ; this determines a decomposition group $D_p = { {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}_p|{\mathbf {Q}}_p)} \hookrightarrow { {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}|{\mathbf {Q}})}$ and its inertia subgroup $I_p = {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}_p|{\mathbf {Q}}_p^{\mathrm {nr}})$ . The inertia subgroup has a descending filtration by its (normal) higher ramification subgroups, the first of which is the wild ramification subgroup $I_p^{\mathrm {w}}$ , the unique (hence normal) Sylow pro-p subgroup of $I_p$ . The quotient $I_p/I_p^{\mathrm {w}}$ is the tame inertia group $I_p^{\mathrm {t}}$ , which is canonically isomorphic to the pro-cyclic $p'$ -group

$$\begin{align*}\varprojlim_{\mathbf{N}_{{\mathbf{F}}|}} {\mathbf{F}}^\times \cong {\hat{{\mathbf{Z}}}}^{(p')}(1) \simeq \prod_{\ell \neq p} {\mathbf{Z}}_\ell(1), \end{align*}$$

where the limit is taken over all finite fields ${\mathbf {F}}$ of characteristic p, the transition maps in the projective limit are the field norms and ${\mathbf {Z}}_\ell (1)$ is the Galois module $\varprojlim _n \mu _{\ell ^n}({\overline {{\mathbf {Q}}}}_p)$ . (See [Reference Serre98, § 1.3]. Note that Serre uses $I_p$ to denote $I_p^{\mathrm {w}}$ , $I_t$ to denote $I_p^{\mathrm {t}}$ and I to denote $I_p$ .)

Lemma 2.7. The absolute Galois group of the residue field ${\mathbf {F}}_p$ , which is canonically isomorphic to ${\hat {{\mathbf {Z}}}}\operatorname {Frob}_p$ , acts on $I_p^{\mathrm {t}}$ via conjugation. One has

$$\begin{align*}\operatorname{Frob}_p x \operatorname{Frob}_p^{-1} = x^p \quad \text{for } x \in I_p^{\mathrm{t}} \,. \end{align*}$$

Note that we have written $I_p^{\mathrm {t}}$ multiplicatively here.

See [Reference Serre98, § 1.7] for the following definition.

Definition 2.8. Let $k \ge 1$ . We define the character $\psi _k$ of $I_p^{\mathrm {t}}$ via the canonical projection from the projective limit as

$$\begin{align*}\psi_k \colon I_p^{\mathrm{t}} \xrightarrow{\sim} \varprojlim_{\mathbf{N}_{{\mathbf{F}}|}} {\mathbf{F}}^\times \twoheadrightarrow {\mathbf{F}}_{p^k}^\times \,. \end{align*}$$

One has $\mathbf {N}_{{\mathbf {F}}_{p^k}|{\mathbf {F}}_p} \circ \psi _k = \psi _1 = \chi _p$ .

The k fundamental characters of level k are the powers $\psi _k^{p^n}$ for $0 \le n < k$ (equivalently, $\psi _k$ followed by the k automorphisms of ${\mathbf {F}}_{p^k}$ ).

2.2 General set-up and notation

In the following, f will always denote a newform of weight $2$ , level N and trivial nebentypus. We let $\mathcal {N}(N)$ denote the set of such newforms; $\mathcal {N}(N, g)$ denotes the subset consisting of forms whose Galois orbit has size g. The Fourier coefficients of f will be denoted $a_n$ (or $a_n(f)$ if we want to make the dependence on f explicit); they generate the coefficient ring ${\mathbf {Z}}[f]$ , which is an order in a totally real number field $F = {\mathbf {Q}}(f)$ (of degree g when $f \in \mathcal {N}(N, g)$ ).

Further, A will denote an abelian variety over ${\mathbf {Q}}$ that is ${\mathbf {Q}}$ -isogenous to $A_f$ (e.g., $A = A_f$ or $A = A' = A_f/A_f[\mathfrak {f}(\mathcal {O}_F/{\mathbf {Z}}[f])]$ as in remark 2.3) and has endomorphism ring $\mathcal {O}$ . Let $\mathfrak {p}$ be a regular prime ideal of $\mathcal {O}$ ; then $\mathfrak {p}$ is a maximal ideal of $\mathcal {O}$ ; we write ${\mathbf {F}}_{\mathfrak {p}} = \mathcal {O}/\mathfrak {p}$ for its residue class field and $p(\mathfrak {p})$ for the residue characteristic (i.e., the characteristic of ${\mathbf {F}}_{\mathfrak {p}}$ ). Then $\rho _{\mathfrak {p}} = \rho _{\mathfrak {p},A}$ is the Galois representation on $A[\mathfrak {p}]$ ; its semi-simplification $\rho _{\mathfrak {p}}^{\mathrm {ss}}$ is independent of the choice of A (as long as $\mathfrak {p}$ is regular). Since we are mostly interested in determining when $\rho _{\mathfrak {p}}$ is irreducible (or has maximal image), knowing $\rho _{\mathfrak {p}}^{\mathrm {ss}}$ is usually enough, and so we suppress the dependency on A in the notation. (Note that when $\mathfrak {p}$ is a prime ideal of $\mathcal {O}_F$ that does not correspond to a regular prime ideal of $\mathcal {O}$ , then $\rho _{\mathfrak {p},A'}$ will be reducible, since there is an isogeny $A' \to A$ whose kernel has nontrivial intersection with $A'[\mathfrak {p}]$ .)

2.3 Determination of the maximal image

One of our goals is to show that the image of $\rho _{\mathfrak {p}}$ is as large as possible for all but finitely many prime ideals $\mathfrak {p}$ (with a small explicit set of possible exceptions). The first step is to determine what this maximal image is. Then we will show that it suffices to consider the image in $\operatorname {PGL}_2({\mathbf {F}}_{\mathfrak {p}})$ .

To show that the projective image is maximal, we have to exclude the possibility that it is contained in one of the maximal subgroups of the maximal projective image, so we need a classification of these maximal subgroups. This will be done in Section 2.4 below.

Definition 2.9. We write $G_{\mathfrak {p}} := \rho _{\mathfrak {p}}({ {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}|{\mathbf {Q}})}) \subseteq \mathrm {GL}_2({\mathbf {F}}_{\mathfrak {p}})$ for the image of $\rho _{\mathfrak {p}}$ . theorem 2.6 implies that $\det (G_{\mathfrak {p}}) = {\mathbf {F}}_p^\times $ , since $\det \circ \rho _{\mathfrak {p}}$ is the mod-p cyclotomic character. We set

$$\begin{align*}G_{\mathfrak{p}}^{{\operatorname{max}}} := \{M \in \mathrm{GL}_2({\mathbf{F}}_{\mathfrak{p}}) : \det(M) \in {\mathbf{F}}_p^\times\} \,; \end{align*}$$

then $G_{\mathfrak {p}} \subseteq G_{\mathfrak {p}}^{{\operatorname {max}}}$ .

We write $\operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}}) := {\mathbf {P}} {\mathrm {SL}}_2({\mathbf {F}}_{\mathfrak {p}})$ for the quotient of $ {\mathrm {SL}}_2({\mathbf {F}}_{\mathfrak {p}})$ by its center $\{\pm I_2\}$ . Note that this is not the same as the group of ${\mathbf {F}}_{\mathfrak {p}}$ -points of the algebraic group $\operatorname {PSL}_2$ . When p is odd, $\operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}})$ has index $2$ in $\operatorname {PGL}_2({\mathbf {F}}_{\mathfrak {p}})$ and is the kernel of $\overline {\det }$ .

Proof. First assume that $\#{\mathbf {F}}_{\mathfrak {p}}> 3$ and let ${{\mathscr {D}H}}$ denote the derived subgroup of a group H. Since the center of $\mathrm {GL}_2({\mathbf {F}}_{\mathfrak {p}})$ is abelian, the assumption ${\mathbf {P}} G = {\mathbf {P}} G_{\mathfrak {p}}^{{\operatorname {max}}}$ implies that ${{\mathscr {D}G}} = {{\mathscr {D}G}}_{\mathfrak {p}}^{{\operatorname {max}}} = {\mathrm {SL}}_2({\mathbf {F}}_{\mathfrak {p}})$ , so $ {\mathrm {SL}}_2({\mathbf {F}}_{\mathfrak {p}}) \le G$ . The second equality follows from

$$\begin{align*}\operatorname{\mathrm{SL}}_2({\mathbf{F}}_{\mathfrak{p}}) = {{\mathscr{D}\operatorname{\mathrm{SL}}}}_2({\mathbf{F}}_{\mathfrak{p}}) \le {{\mathscr{D}G}}_{\mathfrak{p}}^{{\operatorname{max}}} \le {{\mathscr{D}\mathrm{GL}}}_2({\mathbf{F}}_{\mathfrak{p}}) \le \operatorname{\mathrm{SL}}_2({\mathbf{F}}_{\mathfrak{p}}), \end{align*}$$

where the first equality follows from the fact that $\operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}})$ is nonsolvable simple (since nonabelian when $\#{\mathbf {F}}_{\mathfrak {p}}> 3$ ) by [Reference Lang65, Theorem 8.4]. Since both groups map onto ${\mathbf {F}}_p^\times $ under the determinant, we then have exact sequences

$$\begin{align*}1 \to \operatorname{\mathrm{SL}}_2({\mathbf{F}}_{\mathfrak{p}}) \to G \stackrel{\det}{\to} {\mathbf{F}}_p^\times \to 1 \quad\text{and}\quad 1 \to \operatorname{\mathrm{SL}}_2({\mathbf{F}}_{\mathfrak{p}}) \to G_{\mathfrak{p}}^{{\operatorname{max}}} \stackrel{\det}{\to} {\mathbf{F}}_p^\times \to 1, \end{align*}$$

so $\#G = \# {\mathrm {SL}}_2({\mathbf {F}}_{\mathfrak {p}}) \#{\mathbf {F}}_p^\times = \#G_{\mathfrak {p}}^{{\operatorname {max}}}$ , whence the claim.

The two cases ${\mathbf {F}}_{\mathfrak {p}} = {\mathbf {F}}_2$ or ${\mathbf {F}}_3$ can be checked by an easy computation.

2.4 Classification of the maximal subgroups of ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}}$

By lemma 2.11, ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}} = \operatorname {PGL}_2({\mathbf {F}}_{\mathfrak {p}})$ when $\deg \mathfrak {p}$ is odd, and ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}} = \operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}})$ when $\deg \mathfrak {p}$ is even. By proposition 2.12, we know that $G_{\mathfrak {p}} = G_{\mathfrak {p}}^{\operatorname {max}}$ if and only if ${\mathbf {P}}{G_{\mathfrak {p}}} = {\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}}$ , which is equivalent to ${\mathbf {P}}{G_{\mathfrak {p}}} \not \subseteq \Gamma $ for every maximal subgroup $\Gamma $ of ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}}$ . In this section, we recall the classification of these maximal subgroups.

We begin with the case $\deg \mathfrak {p}$ even, where ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}} = \operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}})$ .

When $q = 4$ , the sub-line and normalizer of a split Cartan case are in the same conjugacy class. When $q = 9$ , the normalizers of split Cartan subgroups are contained in exceptional subgroups of type $A_5$ , and the normalizers of nonsplit Cartan subgroups are contained in sub-line stabilizers.

When $\deg \mathfrak {p}$ is odd, we have ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}} = \operatorname {PGL}_2({\mathbf {F}}_{\mathfrak {p}})$ . Since $\det (G_{\mathfrak {p}}) = {\mathbf {F}}_p^\times $ contains elements of ${\mathbf {F}}_{\mathfrak {p}}^\times $ that are non-squares in this case, we also know that ${\mathbf {P}}{G_{\mathfrak {p}}}$ is not contained in $\operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}})$ .

The action of $G_{\mathfrak {p}}$ is reducible if and only if $\rho _{\mathfrak {p}}$ is Borel. In the sub-line case, the invariant sub-line can be the image of a nontrivial invariant subspace of $A[\mathfrak {p}]$ considered as an ${\mathbf {F}}_p$ -vector space. In this case (if $\rho _{\mathfrak {p}}$ is not also Borel), $\rho _{\mathfrak {p}}$ is irreducible as a $2$ -dimensional ${\mathbf {F}}_{\mathfrak {p}}$ -representation, but reducible as a $2 (\deg \mathfrak {p})$ -dimensional ${\mathbf {F}}_p$ -representation. See Section 2.10 below for a more detailed discussion.

2.5 Irreducibility and maximality criteria for fixed $\mathfrak {p}$

In this section, we collect some criteria that allow us to verify that $\rho _{\mathfrak {p}}$ is irreducible or maximal for a given prime ideal $\mathfrak {p}$ , using information from the characteristic polynomials of $\rho _{\mathfrak {p}}(\operatorname {Frob}_\ell )$ for $\ell \nmid Np$ . Recall from theorem 2.6 that the characteristic polynomial of $\rho _{\mathfrak {p}}(\operatorname {Frob}_\ell )$ has the form

$$\begin{align*}T^2 - \bar{a}_\ell T + \bar{\ell}, \end{align*}$$

where $x \mapsto \bar {x}$ denotes the reduction homomorphism ${\mathbf {Z}}[f] \to {\mathbf {F}}_{\mathfrak {p}}$ .

We define some invariants associated to elements of $\operatorname {PGL}_2({\mathbf {F}}_{\mathfrak {p}})$ ; see [Reference Serre98, § 2]. For ${\mathbf {F}}$ a finite field of odd characteristic, we define the Legendre symbol for $a \in {\mathbf {F}}$ as usual:

$$\begin{align*}\Bigl(\frac{a}{{\mathbf{F}}}\Bigr) = \begin{cases} 0 & \text{if }a = 0, \\ 1 & \text{if } a = b^2 \text{ for some } b \in {\mathbf{F}}^\times, \\ -1 & \text{otherwise.} \end{cases} \end{align*}$$

We now assume that the characteristic of ${\mathbf {F}}$ is odd.

If $u(g) \neq 0$ (equivalently, $\operatorname {Tr}(M) \neq 0$ , where M is a lift of g to $\mathrm {GL}_2({\mathbf {F}})$ ), then $\Delta (g) = \left (\frac {u(g)(u(g)-4)}{{\mathbf {F}}}\right )$ . If $u(g) = 0$ , then $\Delta (g) = \left (\frac {-\det (M)}{{\mathbf {F}}}\right ) \neq 0$ , so $\Delta (g) = 0$ is equivalent to $u(g) = 4$ .

We note that $\Delta (g)$ gives the square class of the discriminant of the characteristic polynomial of any lift M of g to $\mathrm {GL}_2({\mathbf {F}})$ . This implies that $\Delta (g) \neq 0$ if and only if M has distinct eigenvalues (and hence is semi-simple). The eigenvalues are in ${\mathbf {F}}$ when $\Delta (g) = 1$ and in the quadratic extension of ${\mathbf {F}}$ and conjugate when $\Delta (g) = -1$ . It follows that the elements of any Borel subgroup of $\operatorname {PGL}_2({\mathbf {F}})$ have $\Delta \neq -1$ . We therefore obtain the following.

This gives a method to prove the irreducibility of $\rho _{\mathfrak {p}}$ by computing

$$\begin{align*}\Delta({\mathbf{P}} \rho_{\mathfrak{p}}(\operatorname{Frob}_\ell)) = \Bigl(\frac{a_\ell^2 - 4 \ell}{{\mathbf{F}}_{\mathfrak{p}}}\Bigr) \end{align*}$$

for a number of primes $\ell \nmid N p(\mathfrak {p})$ . If we obtain the value $-1$ for one such $\ell $ , this shows that $\rho _{\mathfrak {p}}$ is irreducible.

We can use the invariant $u(g)$ to obtain information on the order of g. (See [Reference Serre98, § 2.6 iii].)

We can use this to show that $\rho _{\mathfrak {p}}$ is not exceptional since the elements of $S_4$ have order at most $4$ and the elements of $A_5$ have order $5$ or at most $3$ . So if we can find an element $g \in {\mathbf {P}}{G_{\mathfrak {p}}}$ of order at least $5$ , then ${\mathbf {P}}{G_{\mathfrak {p}}} \not \subseteq S_4$ , and if we can find an element of order $4$ or at least $6$ , then ${\mathbf {P}}{G_{\mathfrak {p}}} \not \subseteq A_5$ .

We now want to rule out the other possible maximal subgroups.

If $\rho _{\mathfrak {p}}$ is dihedral, then ${\mathbf {P}}{G_{\mathfrak {p}}}$ is contained in the normalizer $N(C)$ either of a split or of a nonsplit Cartan subgroup C. The elements of $N(C) \setminus C$ have order $2$ ; hence, $u = 0$ . If C is split, then the nontrivial elements of C have $\Delta = 1$ ; if C is nonsplit, its nontrivial elements have $\Delta = -1$ . So if we find elements in ${\mathbf {P}}{G_{\mathfrak {p}}}$ with $\Delta = 1$ and $u \neq 0$ and also elements with $\Delta = -1$ and $u \neq 0$ , then ${\mathbf {P}}{G_{\mathfrak {p}}}$ cannot be dihedral.

For the sub-line case, we restrict to $\deg \mathfrak {p} \le 2$ . If $\rho _{\mathfrak {p}}$ is sub-line, we must then have $\deg \mathfrak {p} = 2$ , and ${\mathbf {P}}{G_{\mathfrak {p}}} \subseteq \operatorname {PGL}_2({\mathbf {F}}_p)$ (up to conjugation in $\operatorname {PGL}_2({\mathbf {F}}_{\mathfrak {p}})$ ). Since clearly $u(g) \in {\mathbf {F}}_p$ for each element $g \in \operatorname {PGL}_2({\mathbf {F}}_p) \subset \operatorname {PSL}_2({\mathbf {F}}_{\mathfrak {p}})$ , we can exclude the sub-line case when we find an element $g \in {\mathbf {P}}{G_{\mathfrak {p}}}$ such that $u(g) \in {\mathbf {F}}_{\mathfrak {p}} \setminus {\mathbf {F}}_p$ . Without the restriction on $\deg \mathfrak {p}$ , we can similarly exclude the sub-line case when we find $g \in {\mathbf {P}}{G_{\mathfrak {p}}}$ such that ${\mathbf {F}}_p(u(g)) = {\mathbf {F}}_{\mathfrak {p}}$ . It is also the case that the discriminant of the characteristic polynomial of any element is in ${\mathbf {F}}_p$ (up to squares in ${\mathbf {F}}_{\mathfrak {p}}^\times $ ) and therefore a square in ${\mathbf {F}}_{\mathfrak {p}}$ , so that $\Delta \in \{0, 1\}$ . So, similar to the Borel case, this case can also be ruled out as soon as we find an element with $\Delta = -1$ . (This last argument is specific to $\deg \mathfrak {p} = 2^n$ for some n.)

Assuming that we already know that the image is not exceptional, we can therefore prove that it is maximal by considering primes $\ell \nmid Np$ , computing

$$\begin{align*}\Delta(\ell) := \Delta({\mathbf{P}}\rho_{\mathfrak{p}}(\operatorname{Frob}_\ell)) = \Bigl(\frac{a_\ell^2 - 4 \ell}{{\mathbf{F}}_{\mathfrak{p}}}\Bigr) \end{align*}$$

until we have found one $\ell $ such that $\Delta (\ell ) = -1$ and $\mathfrak {p} \nmid a_\ell $ and another $\ell $ such that $\Delta (\ell ) = 1$ and $\mathfrak {p} \nmid a_\ell $ . (Recall that $u({\mathbf {P}}\rho _{\mathfrak {p}}(\operatorname {Frob}_\ell )) \neq 0 \iff \mathfrak {p} \nmid a_\ell $ ).

We obtain the following algorithm that returns a set of possible types of subgroups of ${\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}}$ that can contain ${\mathbf {P}}{G_{\mathfrak {p}}}$ . If this set is empty, then $\rho _{\mathfrak {p}}$ has maximal image. We assume $g = 2$ here since the discussion of the sub-line case above was assuming $\deg \mathfrak {p} \le 2$ , and $g = 2$ is our main case of interest. The algorithm can be modified to work for general g if desired.

We use the symbols $S_4$ and $A_5$ to denote subgroups isomorphic to the respective groups, R (‘reducible’) for a Borel subgroup, L for a sub-line stabilizer, and $N_s$ and $N_{ns}$ for the normalizers of a split or nonsplit Cartan subgroup.

Algorithm 2.19.

The correctness of the algorithm follows from the classification results in Section 2.4 and the discussion in this section. The fact that $N_{ns}$ can be excluded when $\deg \mathfrak {p} = 2$ in Step 3c follows from corollary 2.24 below.

If the image is indeed maximal, then ${\mathbf {P}}{G_{\mathfrak {p}}} = {\mathbf {P}}{G_{\mathfrak {p}}^{\operatorname {max}}}$ contains elements with $u \neq 0$ and $\Delta = 1$ , with $u \neq 0$ and $\Delta = -1$ , of order $\ge 6$ (when $p \ge 5$ ) and of order $4$ . Chebotarëv’s density theorem then guarantees that suitable primes $\ell $ exist to rule out all the possible types. Using an effective version of the density theorem would give an explicit bound B for the primes $\ell $ that have to be considered in the algorithm to be able to decide whether $\rho _{\mathfrak {p}}$ has maximal image. This bound will be too large to be useful in practice, however.

2.6 Explicit computation

When algorithm 2.19 returns a nonempty set of types, we can try to determine the image explicitly as follows. We assume that we have given a curve X of genus $2$ over ${\mathbf {Q}}$ whose Jacobian J is isogenous to $A_f$ ; we will determine the image of $\rho _{J,\mathfrak {p}}$ (which has the same semi-simplification as $\rho _{\mathfrak {p},f}$ ), assuming that $\mathfrak {p}$ is a regular prime of $\mathrm {End}_{{\mathbf {Q}}}(J)$ .

We compute (using Magma, say) the big period matrix associated to J, which allows us to write $J({\mathbf {C}}) = {\mathbf {C}}^2/\Lambda $ for some (numerically) explicit lattice $\Lambda $ . We can also determine the action of $\mathrm {End}(J)$ on ${\mathbf {C}}^2$ , and so we can approximate numerically the points in $J({\mathbf {C}})[\mathfrak {p}]$ . We represent these points by (numerical) divisors on X, which we then recognize as divisors supported in algebraic points (this will work when the precision is sufficiently large). We then verify that the algebraic points on J we obtain are indeed in $J[\mathfrak {p}]$ . Knowing the points explicitly as algebraic points allows us to determine the Galois action.

Since Magma can easily determine the torsion subgroup of $J({\mathbf {Q}})$ using the algorithm described in [Reference Stoll111, § 11], we can at least deduce that the representation associated to some prime ideal $\mathfrak {p}$ with $p(\mathfrak {p}) = p$ is reducible if $J({\mathbf {Q}})[p]$ is nontrivial.

2.7 The image of inertia at p

Our next goal will be to prove that $\rho _{\mathfrak {p}}$ is irreducible (or even maximal) for all but finitely many $\mathfrak {p}$ , with a small explicit set of possible exceptions. To this end, we need to study the representations $\rho _{\mathfrak {p}}$ more carefully, so that we can extract some uniform statements. We begin by considering the action of the inertia group at the prime p. Recall the definitions and the notations $I_p$ , $I_p^{\mathrm {w}}$ , $I_p^{\mathrm {t}}$ from Section 2.1. Also recall the fundamental characters $\psi _k$ from definition 2.8. In the following, g is arbitrary again.

We now consider $\rho _{\mathfrak {p}}|_{I_p}$ , where $p = p(\mathfrak {p})$ is the residue characteristic of $\mathfrak {p}$ . We have the following result.

Corollary 2.26. Let $\mathfrak {p}$ be a regular prime of ${\mathbf {Z}}[f]$ of residue characteristic p and assume that $\rho _{\mathfrak {p}}$ is reducible and $p^2 \nmid N$ . Then there is a character $\varepsilon $ of ${ {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}|{\mathbf {Q}})}$ with values in ${\mathbf {F}}_{\mathfrak {p}}^\times $ and conductor d such that $d^2 \mid N$ and (with respect to a suitable basis)

$$\begin{align*}\rho_{\mathfrak{p}} = \begin{pmatrix} \varepsilon \chi_p^n & * \\ 0 & \varepsilon^{-1} \chi_p^{1-n} \end{pmatrix} \end{align*}$$

with $n = 0$ or $n = 1$ .

Proof. The semi-simplification of $\rho _{\mathfrak {p}}$ splits as a direct sum $\pi _1 \oplus \pi _2$ of one-dimensional characters $\pi _1, \pi _2 \colon { {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}|{\mathbf {Q}})} \to {\mathbf {F}}_{\mathfrak {p}}^\times $ . By lemma 2.25, $\pi _1|_{I_p} = \chi _p^n$ for some n, and so $\pi _2|_{I_p} = \chi _p^{1-n}$ . We can therefore write $\pi _1 = \varepsilon \chi _p^n$ with some character $\varepsilon $ that is unramified at p. Since $\chi _p = \det \circ \rho _{\mathfrak {p}} = \pi _1 \cdot \pi _2$ , it follows that $\pi _2 = \varepsilon ^{-1} \chi _p^{1-n}$ . We then have

$$\begin{align*}d^2 = \operatorname{cond}(\varepsilon) \operatorname{cond}(\varepsilon^{-1}) \mid \operatorname{cond}(\rho_{\mathfrak{p}}) \mid N \,. \end{align*}$$

Since $p^2 \nmid N$ , we have $n \in \{0, 1\}$ by theorem 2.21.

We now consider the case that $\rho _{\mathfrak {p}}({ {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}_p|{\mathbf {Q}}_p)})$ is contained in the normalizer of a Cartan subgroup of $\mathrm {GL}_2({\mathbf {F}}_{\mathfrak {p}})$ .

2.8 The image of inertia at a prime $\ell \neq p$

We now consider the image $\rho _{\mathfrak {p}}(I_\ell )$ of the inertia subgroup at a prime $\ell \neq p$ .

Proof. If $\ell \nmid N$ , then $\rho _{\mathfrak {p}}$ is unramified at $\ell $ , and so the image of inertia is trivial. Otherwise, since the prime-to-p part of the Artin conductor of $\rho _{\mathfrak {p}}$ divides the prime-to-p part of N, it follows from $v_\ell (N) = 1$ that the image of wild inertia is trivial and that the image of inertia has a one-dimensional fixed subspace; see [Reference Serre97, p. 181]. So

$$\begin{align*}\rho_{\mathfrak{p}}|_{I_\ell} \sim \begin{pmatrix} 1 & * \\ 0 & \chi_p \end{pmatrix} \,. \end{align*}$$

Since $\chi _p$ is unramified at $\ell $ , this implies that $\rho _{\mathfrak {p}}(I_\ell )$ is unipotent. (See also [Reference Ribet90, Section 2]; note that the definition of the conductor is purely local.)

Corollary 2.34. Let $\mathfrak {p}$ be a regular prime ideal of ${\mathbf {Z}}[f]$ of residue characteristic $p> 3$ and assume that $G_{\mathfrak {p}}$ is contained in the normalizer $N(C)$ of a Cartan subgroup C. Since C has index $2$ in $N(C)$ , we obtain a quadratic character (which can be trivial)

$$\begin{align*}\varepsilon_{\mathfrak{p}} \colon {\operatorname{\mathrm{Gal}}({\overline{{\mathbf{Q}}}}|{\mathbf{Q}})} \stackrel{\rho_{\mathfrak{p}}}{\to} N(C) \to N(C)/C \simeq \{\pm 1\} \,; \end{align*}$$

let d be its conductor. Then the odd part of $d^2$ divides N. Moreover, if $4 \nmid N$ , then $d^2 \mid N$ .

2.9 Explicit irreducibility for almost all $\mathfrak {p}$

It is known that $\rho _{\mathfrak {p}}$ is irreducible for all but finitely many $\mathfrak {p}$ . Lombardo [Reference Lombardo69, Thm. 1.4] gives an explicit (but very large) bound for the reducible primes (actually, for the primes such that the image is not maximal). What we need, however, is to determine as exactly as possible the finite set of primes $\mathfrak {p}$ such that $\rho _{\mathfrak {p}}$ is reducible in each concrete case.

In view of corollary 2.26, we make the following definition.

We then obtain the following criterion. (Compare [Reference Dieulefait38, § 3.1], where similar criteria are used to show that the mod-p Galois representations associated to an abelian surface with minimal endomorphism ring are maximal for almost all primes p.)

Proposition 2.38. Let $\ell \nmid N$ be a prime and let $m(\ell )$ be the order of $\ell $ in $({\mathbf {Z}}/d_{\operatorname {max}}{\mathbf {Z}})^\times $ . Let $\mathfrak {p}$ be a regular prime of ${\mathbf {Z}}[f]$ of residue characteristic p. If $p^2 \nmid N$ and

$$\begin{align*}\mathfrak{p} \nmid \ell \cdot \operatorname{res}_T(T^2 - a_\ell T + \ell, T^{m(\ell)} - 1) \in {\mathbf{Z}}[f], \end{align*}$$

then $\rho _{\mathfrak {p}}$ is irreducible.

We note that when $\varphi (d_{\operatorname {max}}) = 1$ (which is the case, for example, when N is squarefree) or $\ell \equiv 1 \bmod d_{\operatorname {max}}$ , the resultant simplifies to $\ell + 1 - a_\ell $ .

Proof. Let us prove the contrapositive. Thus, suppose that $\rho _{\mathfrak {p}}$ is reducible. Let $\varepsilon $ be the character in corollary 2.26. It can be considered as a Dirichlet character of conductor $d \mid d_{\operatorname {max}}$ with values in ${\mathbf {F}}_{\mathfrak {p}}^\times $ . Since $p^2 \nmid N$ , we have $n = 0$ or $n = 1$ in corollary 2.26, and by symmetry (and since what we do depends only on the semi-simplification of $\rho _{\mathfrak {p}}$ ), we can assume $n = 0$ . If $\ell = p$ , then $\mathfrak {p} \mid \ell $ , and we are done. Hence, suppose $\ell \nmid Np$ . Then $\rho _{\mathfrak {p}}(\operatorname {Frob}_\ell )$ is well-defined and has $\varepsilon (\ell )$ as an eigenvalue, which is also a root of unity of order dividing $m(\ell )$ . So the characteristic polynomial $T^2 - \bar {a}_\ell T + \bar {\ell } \in {\mathbf {F}}_{\mathfrak {p}}[T]$ of $\rho _{\mathfrak {p}}(\operatorname {Frob}_\ell )$ has at least one root in common with $T^{m(\ell )} - 1$ . This is equivalent to

$$\begin{align*}\operatorname{res}_T(T^2 - \bar{a}_\ell T + \bar{\ell}, T^{m(\ell)} - 1) = 0 \,. \end{align*}$$

Since the resultant is compatible with ring homomorphisms, this implies that $\mathfrak {p}$ divides the resultant in the statement.

Remark 2.39. One can alternatively consider the condition

$$\begin{align*}p \mid \ell \cdot \operatorname{res}_T\bigl(\operatorname{charpol}(\rho_{p^\infty}(\operatorname{Frob}_\ell))(T), T^{m(\ell)} - 1\bigr) \in {\mathbf{Z}} \end{align*}$$

with the same $m(\ell )$ as above. Using the fact that the resultant is the product of all differences of roots of the first and of the second polynomial, together with the Weil conjectures for the characteristic polynomial of $\operatorname {Frob}_\ell $ on $T_p A$ , we see that the resultant above is an integer R satisfying

$$\begin{align*}0 < (\sqrt{\ell} - 1)^{2gm(\ell)} < R < (\sqrt{\ell}+1)^{2gm(\ell)} \,. \end{align*}$$

In particular, it is nonzero, so taking just one $\ell \nmid N$ gives a relatively small bound for the set of primes p such that $\rho _{\mathfrak {p}}$ is reducible for some $\mathfrak {p} \mid p$ . In practice, one takes several $\ell $ and uses the gcd of the resultants to obtain reasonably sharp bounds.

This leads to the following algorithm. Recall the notation $p(\mathfrak {p})$ for the residue characteristic of the prime ideal $\mathfrak {p}$ .

Algorithm 2.40.

We can then use algorithm 2.19 on the regular odd primes in S to try to show that $\rho _{\mathfrak {p}}$ is irreducible even though algorithm 2.40 was unable to prove that. One can also use the idea from remark 2.27.

Example 2.42. Let $A = J_0(35)^{w_7}$ be the Jacobian of the modular curve quotient $X_0(35)/\langle w_7 \rangle $ , where $w_7$ denotes the Atkin–Lehner involution. A corresponds to the Galois orbit of size 2 of newforms of level 35, weight $2$ and trivial nebentypus. Let f be one of these two newforms. Then ${\mathbf {Z}}[f]$ is the maximal order of ${\mathbf {Q}}(\sqrt {17})$ . Since the level $N = 35$ is squarefree, $d_{\operatorname {max}} = 1$ .

Let $\beta = (1 + \sqrt {17})/2$ be a generator of ${\mathbf {Z}}[f]$ . We then have $a_2 = -\beta $ (for one of the two conjugate newforms), so the resultant is

$$\begin{align*}\operatorname{res}_T(T^2 + \beta T + 2, T - 1) = 3 + \beta, \end{align*}$$

which is an element of norm $2^3$ . This shows that $\rho _{\mathfrak {p}}$ is irreducible for all prime ideals $\mathfrak {p}$ with odd residue characteristic. We also have $a_3 = \beta - 1$ , and the corresponding resultant is

$$\begin{align*}\operatorname{res}_T(T^2 - (\beta - 1) T + 3, T - 1) = 5 - \beta, \end{align*}$$

whose norm is $2^4$ and which is not divisible by $2$ , so it generates a power of one of the two prime ideals above $2$ in ${\mathbf {Z}}[f]$ ; explicitly,

$$\begin{align*}\langle 5 - \beta \rangle = \langle \beta + 1 \rangle^4 = \mathfrak{p}^4 \,. \end{align*}$$

Write $\langle 2 \rangle = \mathfrak {p} \mathfrak {p}'$ . Then we can deduce that $\rho _{\mathfrak {p}'}$ is irreducible.

Magma computes that $A({\mathbf {Q}})_{\mathrm {tors}} \cong {\mathbf {Z}}/16{\mathbf {Z}}$ . This shows that $\rho _{\mathfrak {p}}$ must be reducible (and that it is nontrivial since $A[\mathfrak {p}]({\mathbf {Q}}) \cong {\mathbf {Z}}/2{\mathbf {Z}}$ and not $({\mathbf {Z}}/2{\mathbf {Z}})^2$ ).

Example 2.43. We now consider $A = J_0(125)^+$ , which corresponds to a Galois orbit of newforms with coefficient ring the maximal order of ${\mathbf {Q}}(\sqrt {5})$ . Let $\alpha = (1 + \sqrt {5})/2$ . Then we can pick one of the newforms f such that $a_2 = -\alpha $ and $a_3 = \alpha - 2$ . Here, $d_{\operatorname {max}} = 5$ and so $m(2) = m(3) = 4$ . We find that

$$\begin{align*}\begin{array}{r@{{}:={}}l@{{}={}}l} r_2 & \operatorname{res}_T(T^2 + \alpha T + 2, T^4 - 1) & 15 + 5 \alpha \quad\text{and} \\ r_3 & \operatorname{res}_T(T^2 - (\alpha-2) T + 3, T^4 - 1) & 90 - 15 \alpha \,. \end{array} \end{align*}$$

The ideal of ${\mathbf {Z}}[f]$ generated by $2 r_2$ and $3 r_3$ has norm $5^2$ . So the algorithm shows that $\rho _{\mathfrak {p}}$ is irreducible for all primes $\mathfrak {p} \neq \langle \sqrt {5} \rangle $ . An explicit computation shows that $\rho _{\langle \sqrt {5} \rangle }$ is reducible; see example 2.20, (1).

2.10 Excluding the sub-line and exceptional cases for almost all $\mathfrak {p}$

In the following, we require that the coefficient field of the newform is of degree $2$ , and so $\dim A = 2$ as well. This simplifies the discussion of the sub-line case.

Our next goal will be to show that $\rho _{\mathfrak {p}}$ has maximal image for all $\mathfrak {p}$ outside a small explicit finite set. Starting from the result of the previous subsection, it remains to show that for all but a few explicit $\mathfrak {p}$ , ${\mathbf {P}}{G_{\mathfrak {p}}}$ is not contained in the stabilizer of a sub-line (when $\deg \mathfrak {p} = 2$ ), in a maximal subgroup of type $S_4$ or $A_5$ , or in the normalizer of a Cartan subgroup.

We begin with the sub-line case and assume that $\rho _{\mathfrak {p}}$ is irreducible and $p(\mathfrak {p})$ is odd. In this case, $G_{\mathfrak {p}} \subseteq {\mathbf {P}}^{-1}(\operatorname {PGL}_2({\mathbf {F}}_p)) \cap G_{\mathfrak {p}}^{\operatorname {max}}$ , which is a group containing $\mathrm {GL}_2({\mathbf {F}}_p)$ with index $2$ . More precisely, let $\alpha \in {\mathbf {F}}_{\mathfrak {p}}^\times \setminus {\mathbf {F}}_p^\times $ be such that $\alpha ^2 \in {\mathbf {F}}_p^\times $ ; then the group is the union of $\mathrm {GL}_2({\mathbf {F}}_p)$ and $\alpha \mathrm {GL}_2({\mathbf {F}}_p)$ . So $G_{\mathfrak {p}}$ stabilizes the union of a $2$ -dimensional ${\mathbf {F}}_p$ -subspace U (that is not a $1$ -dimensional ${\mathbf {F}}_{\mathfrak {p}}$ -subspace) of $A[p]$ and $\alpha U$ . If $G_{\mathfrak {p}} \subseteq \mathrm {GL}_2({\mathbf {F}}_p)$ (for some choice of ${\mathbf {F}}_{\mathfrak {p}}$ -basis of $A[p]$ ), then these two ${\mathbf {F}}_p$ -subspaces are fixed individually, and so $\rho _p$ is the direct sum of two copies of a $2$ -dimensional ${ {\mathrm {Gal}}({\overline {{\mathbf {Q}}}}|{\mathbf {Q}})}$ -representation over ${\mathbf {F}}_p$ ; in particular, $\rho _p$ is reducible. Conversely, if $\rho _p$ is reducible (but $\rho _{\mathfrak {p}}$ is not), then ${\mathbf {P}}{G_{\mathfrak {p}}}$ must be contained in the stabilizer of the sub-line that is the image of a $2$ -dimensional ${\mathbf {F}}_p$ -subspace fixed by $G_{\mathfrak {p}}$ . So, by showing that $\rho _{\mathfrak {p}}$ is irreducible and ${\mathbf {P}}{G_{\mathfrak {p}}}$ is not contained in the stabilizer of a sub-line, we also show that A has no nontrivial isogenies of degree a power of p.

In any case, we know from the discussion in Section 2.5 that the element $u({\mathbf {P}}\rho _{\mathfrak {p}}(\operatorname {Frob}_\ell )) = a_\ell ^2/\ell \in {\mathbf {F}}_{\mathfrak {p}}$ must be in ${\mathbf {F}}_p$ for all $\ell \nmid Np$ if ${\mathbf {P}}{G_{\mathfrak {p}}}$ is contained in the stabilizer of a sub-line. This is equivalent to $p \mid a_\ell ^2 - {a_\ell ^\sigma }^2$ , where $\sigma $ denotes the nontrivial automorphism of ${\mathbf {Z}}[f]$ . Since in our setting, there always are $\mathfrak {p}$ of degree $2$ such that ${\mathbf {P}}{G_{\mathfrak {p}}}$ does not consist entirely of elements g such that $u(g) \in {\mathbf {F}}_p$ , there will be a set of primes $\ell \nmid N$ of positive density such that $a_\ell ^2 \neq {a_\ell ^\sigma }^2$ . (If f has an inner twist by a quadratic character, then $a_\ell = \pm a_\ell ^\sigma $ for almost all $\ell $ . But in this case, $A_f$ is isogenous to the Weil restriction of an elliptic curve over the quadratic field associated to the character, and so $A_f$ is not absolutely simple.) We can replace $a_\ell ^2 - {a_\ell ^\sigma }^2$ by $(a_\ell ^2 - {a_\ell ^\sigma }^2)/\sqrt {\operatorname {disc}({\mathbf {Z}}[f])} \in {\mathbf {Z}}$ (note that the prime divisors of $\operatorname {disc}({\mathbf {Z}}[f])$ are always ramified, so the corresponding primes of ${\mathbf {Z}}[f]$ have degree $1$ ). This leads to the following algorithm.

Algorithm 2.44.

By the discussion above, the algorithm will not return ‘failure’ when B is sufficiently large.

We now consider the case of exceptional image. Our analysis of the projective image of the inertia group $I_p$ allows us to find elements of order at least $6$ in ${\mathbf {P}}{\rho _{\mathfrak {p}}(I_p)} \leq {\mathbf {P}}{G_{\mathfrak {p}}}$ in most cases, which implies that ${\mathbf {P}}{G_{\mathfrak {p}}}$ is not contained in $S_4$ or $A_5$ .

So we only have to consider prime ideals $\mathfrak {p}$ of residue characteristic p such that $p \le 5$ or $p^2 \mid N$ . By the classification in Section 2.4, we can also exclude $p = 3$ when $\deg \mathfrak {p} = 1$ and $p = 2$ . We can then run algorithm 2.19 on these finitely many $\mathfrak {p}$ to reduce the set of possibly exceptional primes further.

2.11 Proving non-CM

Recall that our goal is to show that $G_{\mathfrak {p}} = G_{\mathfrak {p}}^{\operatorname {max}}$ for all $\mathfrak {p}$ outside an explicit small finite set. Now if f has CM, then $G_{\mathfrak {p}}$ will always be contained in the normalizer of a Cartan subgroup, so in this case, our task is impossible. Note that by proposition 2.2, in our case of interest when $g = 2$ , if the associated abelian surface is absolutely simple, f cannot have CM. However, since it may be of interest in other situations, we describe a suitable algorithm.