Theorem 1.1 .

If $p\geqslant 11$ the modular curve $X_{\text{ns}}(p)$ is not hyperelliptic. If $p\geqslant 13$ the modular curve $X_{\text{ns}}^{+}(p)$ is not hyperelliptic.

Theorem 5.2 .

If $p\geqslant 29$ , all the automorphisms of $X_{\text{ns}}(p)$ preserve the cusps. If $p\equiv 1~\text{mod}~12$ and $p\neq 13$ , the automorphism group of $X_{\text{ns}}(p)$ is generated by the modular involution $w$ .

Theorem 5.3 .

Let $p\geqslant 29$ . If there exists an exceptional automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ , then $X_{\text{ns}}^{+}(p)$ has a non-CM rational point.

Proof. (Following [Reference OggOgg74, p. 455, Theorem 3]). Let $q\neq p$ be a prime number, let $\mathbb{F}_{q}$ be the finite field with $q$ elements and let $\overline{\mathbb{F}_{q}}$ be an algebraic closure. The points of $X_{\text{ns}}(p)(\overline{\mathbb{F}_{q}})$ parametrize pairs $(E,\unicode[STIX]{x1D711})$ , where $E$ is an elliptic curve over $\overline{\mathbb{F}_{q}}$ and $\unicode[STIX]{x1D711}$ is an isomorphism from the $p$ -torsion points of $E$ to $\mathbb{Z}/p\mathbb{Z}\times \mathbb{Z}/p\mathbb{Z}$ . Moreover, two pairs $(E,\unicode[STIX]{x1D711})$ , $(E^{\prime },\unicode[STIX]{x1D711}^{\prime })$ are parametrized by the same point of $X_{\text{ns}}(p)(\overline{\mathbb{F}_{q}})$ if and only if there exist an isomorphism $f$ from $E$ to $E^{\prime }$ such that, on the $p$ -torsion of $E$ , we have $\unicode[STIX]{x1D711}^{\prime }\circ f=M\circ \unicode[STIX]{x1D711}$ , where $M$ is a generator of a nonsplit Cartan subgroup $C$ of $\text{GL}_{2}(\mathbb{F}_{p})$ . The Frobenius automorphism $\unicode[STIX]{x1D70E}$ generating $\text{Gal}(\overline{\mathbb{F}_{q}}/\mathbb{F}_{q^{2}})$ acts on $X_{\text{ns}}(p)(\overline{\mathbb{F}_{q}})$ as $\unicode[STIX]{x1D70E}(E,\unicode[STIX]{x1D711})=(E^{\unicode[STIX]{x1D70E}},\unicode[STIX]{x1D711}\circ \unicode[STIX]{x1D70E}^{-1}),$ so that every elliptic curve $E$ over $\mathbb{F}_{q^{2}}$ gives a point $(E,\unicode[STIX]{x1D711})$ in $X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})$ if and only if $\unicode[STIX]{x1D711}\circ \unicode[STIX]{x1D70E}^{-1}\circ \unicode[STIX]{x1D711}^{-1}$ is contained in $C$ .

Take a supersingular elliptic curve $E$ over $\overline{\mathbb{F}_{q}}$ . This means that the $q$ -torsion of $E$ is trivial, that the endomorphism algebra $\text{End}(E)$ is an order in a quaternion algebra, and implies that $E$ can be defined over $\mathbb{F}_{q^{2}}$ [Reference SilvermanSil09, p. 144, Theorem 3.1, (a)]. The fact that multiplication by $q$ is purely inseparable implies that the $q^{2}$ th power Frobenius endomorphism $\unicode[STIX]{x1D711}_{q^{2}}$ and multiplication by $q$ differ by an automorphism of $E$ , which means that $\unicode[STIX]{x1D711}_{q^{2}}=\unicode[STIX]{x1D701}q$ in $\text{End}(E)$ and $\unicode[STIX]{x1D701}$ is a root of unity. If $j\neq 0,1728$ , we have $\unicode[STIX]{x1D711}_{q^{2}}=\pm q$ . If $j=0,1728$ , the elliptic curve $E$ is isomorphic over $\overline{\mathbb{F}_{q}}$ to a curve $E^{\prime }$ defined over $\mathbb{F}_{q}$ with equation $y^{2}+y=x^{3}$ or $y^{2}=x^{3}-x$ . We have $\#E^{\prime }(\mathbb{F}_{q})=q+1$ ([Reference SilvermanSil09, p. 154, Exercise 5.10, (b)] when $q\geqslant 5$ , a direct count when $q=2,3$ ). Hence the $q$ th power Frobenius endomorphism of $E^{\prime }$ has characteristic polynomial $x^{2}+q=0$ which implies that $\unicode[STIX]{x1D711}_{q^{2}}=-q$ in $\text{End}(E)$ .

This shows that every supersingular elliptic curve over $\overline{\mathbb{F}_{q}}$ is isomorphic to a supersingular elliptic curve $E$ defined over $\mathbb{F}_{q^{2}}$ , with the property that $\unicode[STIX]{x1D711}_{q^{2}}$ acts on the $p$ -torsion points as multiplication by $q$ or $-q$ . Which says that $\unicode[STIX]{x1D711}_{q^{2}}$ acts on the $p$ -torsion of $E$ as a scalar matrix of $\text{GL}_{2}(\mathbb{F}_{p})$ and therefore as an element contained in every conjugate of the nonsplit Cartan subgroup $C$ in $\text{GL}_{2}(\mathbb{F}_{p})$ . Hence, $E$ gives a point in $X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})$ for each conjugate of $C$ in $\text{GL}_{2}(\mathbb{F}_{p})$ , up to automorphisms of $E$ , in the sense that a pair $(E,\unicode[STIX]{x1D711})$ gives the same point on $X_{\text{ns}}(p)$ as $(E,\unicode[STIX]{x1D711}\circ f)$ for any automorphism $f$ of $E$ . Among the automorphisms of any supersingular elliptic curve, we have multiplication by $-1$ , which acts on torsion points as an element contained in $C$ . Thus, since the index of a nonsplit Cartan subgroup in $\text{GL}_{2}(\mathbb{F}_{p})$ is equal to $p(p-1)$ , we have the following inequality

$$\begin{eqnarray}\#X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})\geqslant p(p-1)\cdot 2\cdot \mathop{\sum }_{\substack{ E\;\text{over}\;\overline{\mathbb{F}_{q}} \\ \text{supersingular}}}\frac{1}{\#\text{Aut}(E)}=\frac{p(p-1)(q-1)}{12},\end{eqnarray}$$

where in the last equality we used the Deuring–Eichler formula [Reference SilvermanSil09, p. 154, Exercise 5.9]

$$\begin{eqnarray}\mathop{\sum }_{\substack{ E\;\text{over}\;\overline{\mathbb{F}_{q}} \\ \text{supersingular}}}\frac{1}{\#\text{Aut}(E)}=\frac{q-1}{24}.\end{eqnarray}$$

Note that this lower bound on the number of supersingular points can also easily be deduced from [Reference Baker, González-Jiménez, González and PoonenBGJGP05, Lemmas 3.20 and 3.21].

Suppose now that $X_{\text{ns}}(p)$ is hyperelliptic. Then we have

$$\begin{eqnarray}\frac{p(p-1)(q-1)}{12}\leqslant \#X_{\text{ns}}(p)(\mathbb{F}_{q^{2}})\leqslant 2(q^{2}+1),\end{eqnarray}$$

where the second inequality holds because $X_{\text{ns}}(p)$ is hyperelliptic and because the projective line over $\mathbb{F}_{q^{2}}$ has exactly $q^{2}+1$ points. Now, putting $q=2$ we have

$$\begin{eqnarray}\frac{p(p-1)}{12}\leqslant \#X_{\text{ns}}(p)(\mathbb{F}_{4})\leqslant 10\end{eqnarray}$$

and we obtain $p\leqslant 11$ which ends the proof for $X_{\text{ns}}(p)$ in the case $p\geqslant 13$ . With an analogous reasoning, taking into account that the index of the normalizer of a nonsplit Cartan subgroup in $\text{GL}_{2}(\mathbb{F}_{p})$ is equal to $\frac{p(p-1)}{2}$ , we obtain the statement for $X_{\text{ns}}^{+}(p)$ in the case $p\geqslant 17$ .

The case $p=11$ for $X_{\text{ns}}(p)$ and the case $p=13$ for $X_{\text{ns}}^{+}(p)$ can be worked out by explicitly counting the points over $\mathbb{F}_{4}$ of the respective curves. Recall that the Jacobian of $X_{\text{ns}}(p)$ is isogenous to the new part of the Jacobian of $X_{0}(p^{2})$ and the Jacobian of $X_{\text{ns}}^{+}(p)$ is isogenous to the new part of the Jacobian of $X_{0}^{\ast }(p^{2})\stackrel{\text{def}}{=}X_{0}(p^{2})/\langle w_{p^{2}}\rangle$ , where $w_{p^{2}}$ is the Atkin–Lehner involution [Reference ChenChe98, Reference de Smit and EdixhovendSE00]. Then we can count the points over $\mathbb{F}_{4}$ by looking at the roots of the characteristic polynomial of a Frobenius at $2$ on the Jacobian of $X_{0}(p^{2})$ , and using the well- known formula

$$\begin{eqnarray}\#C(\mathbb{F}_{q^{r}})=q^{r}+1-\mathop{\sum }_{i=1}^{2g}\unicode[STIX]{x1D6FC}_{i}^{r}\end{eqnarray}$$

where $C$ is a nonsingular projective curve of genus $g$ over the finite field $\mathbb{F}_{q}$ with $q$ prime, and the $\unicode[STIX]{x1D6FC}_{i}$ ’s are the roots of the characteristic polynomial of the Frobenius automorphism at $q$ , acting on the Jacobian variety of $C$ .

Looking at the tables in [Reference SteinSte12] of weight-2 newforms for $\unicode[STIX]{x1D6E4}_{0}(121)$ , we see that the Jacobian of $X_{\text{ns}}(11)$ is isogenous to the product of four elliptic curves for which the traces of a Frobenius automorphism at 2 are respectively $-1$ , $0$ , $1$ , $2$ , so that the roots of the characteristic polynomial of a Frobenius at 2 acting on the Jacobian of $X_{\text{ns}}(11)$ are $\frac{-1\pm \sqrt{-7}}{2}$ , $\pm \sqrt{-2}$ , $\frac{1\pm \sqrt{-7}}{2}$ , $1\pm i$ . Hence $\#X_{\text{ns}}(11)(\mathbb{F}_{4})=15>10$ .

Now, in the same tables, we look at the weight-2 newforms for $\unicode[STIX]{x1D6E4}_{0}(169)$ . We see that the Jacobian of $X_{\text{ns}}^{+}(13)$ is isogenous to a simple abelian variety $A$ over $\mathbb{Q}$ of dimension $3$ . The eigenvalues of the Hecke operator $T_{2}$ acting on $A$ are the roots $a_{i}$ , with $i=1,2,3$ , of the polynomial $x^{3}+2x^{2}-x-1$ . Then the characteristic polynomial of a Frobenius at 2 is the product

$$\begin{eqnarray}\mathop{\prod }_{i=1}^{3}(x^{2}-a_{i}x+2).\end{eqnarray}$$

It allows us to compute $\#X_{\text{ns}}^{+}(13)(\mathbb{F}_{4})=11>10$ .◻

Proof. [Reference RibetRib75, p. 556, Theorem 1.1]. ◻

Proof. [Reference RibetRib75, p. 557, Theorem 2.3] ◻

Proof. [Reference Deligne and RapoportDR73, p. 286, Theorem 6.9]. ◻

Proof. Proposition 2.2 tells us that endomorphisms of $J_{0}^{H}(p^{2})$ can be defined over a field $K$ which is a compositum of quadratic extensions of $\mathbb{Q}$ . Furthermore, Propositions 2.1 and 2.3 imply that $K$ can be taken unramified outside $p$ . Thus, we can take $K=\mathbb{Q}(\sqrt{p})$ if $p\equiv 1~\text{mod}~4$ and $K=\mathbb{Q}(\sqrt{-p})$ if $p\equiv 3~\text{mod}~4$ .◻

Proof. (Following [Reference Kenku and MomoseKM88, p. 55, Lemma 1.4]) Let $u$ be an automorphism of $X_{\text{ns}}(p)$ and $\unicode[STIX]{x1D70E}$ a element of $\text{Gal}(\overline{\mathbb{Q}}/K(p))$ . We define $v=u^{\unicode[STIX]{x1D70E}}u^{-1}$ and let $d$ be the order of $v$ in the automorphism group of $X_{\text{ns}}(p)$ . We call $Y$ the quotient of $X_{\text{ns}}(p)$ by the automorphism $v$ . Let $g_{Y}$ be the genus of $Y$ . We have that $v$ induces an automorphism of the Jacobian $J_{\text{ns}}(p)\sim J_{\text{ns}}^{H}(p)\times J_{\text{ns}}^{C}(p)$ which acts trivially on $J_{\text{ns}}^{H}(p)$ by Proposition 2.4, because $J_{\text{ns}}^{H}(p)$ is a factor of $J_{0}^{H}(p^{2})$ . Thus we have the following morphisms of abelian varieties with finite kernel

$$\begin{eqnarray}\displaystyle J_{\text{ns}}^{H}(p) & \rightarrow & \displaystyle J_{\text{ns}}^{H}(p)\times J_{\text{ns}}^{C}(p)\rightarrow J_{\text{ns}}(p)\nonumber\\ \displaystyle x & \mapsto & \displaystyle (x,0)\nonumber\end{eqnarray}$$

where the image of the composition is contained in $J_{\text{ns}}(p)^{v}\stackrel{\text{def}}{=}\{D\in J_{\text{ns}}(p):vD=D\}$ . Hence there are morphisms of group varieties

$$\begin{eqnarray}J_{\text{ns}}^{H}(p)\longrightarrow J_{\text{ns}}(p)^{v}\longrightarrow \text{Jac}(Y)\end{eqnarray}$$

where the second map is the push-forward of divisors, and the composition is a morphism of abelian varieties. Furthermore, both maps have finite kernel, hence $g_{Y}\geqslant g_{\text{ns}}^{H}(p)$ and we have by the Riemann–Hurwitz formula

$$\begin{eqnarray}g_{\text{ns}}(p)-1\geqslant d(g_{Y}-1)\geqslant 2g_{\text{ns}}^{H}(p)-2,\end{eqnarray}$$

where in the second inequality we supposed $d>1$ . Now using $g_{\text{ns}}(p)=g_{\text{ns}}^{H}(p)+g_{\text{ns}}^{C}(p)$ we obtain $g_{\text{ns}}(p)\leqslant 2g_{\text{ns}}^{C}(p)+1$ . Since $J_{\text{ns}}(p)$ is isogenous over $\mathbb{Q}$ to the new part of $J_{0}(p^{2})$ we have $g_{\text{ns}}^{C}(p)=g_{0}^{C}(p^{2})$ . Indeed, the old part of $J_{0}(p^{2})$ is isogenous to the product of two copies of the Jacobian of $J_{0}(p)$ and these cannot have abelian subvarieties of CM-type (see [Reference ShimuraShi72, p. 140, Remark 1.7]). Furthermore, $g_{0}^{C}(p^{2})=0$ if $p\equiv 1~\text{mod}~4$ and $g_{0}^{C}(p^{2})=h(-p)$ if $p\equiv 3~\text{mod}~4$ , where $h(-p)$ is the class number of $\mathbb{Q}(\sqrt{-p})$ . Indeed, in the decomposition (2.1) with $N=p^{2}$ , if a factor is of CM-type, then it is the product of copies of an elliptic curve with complex multiplication by an imaginary quadratic field $K$ of discriminant $-p$ . This leaves no possibilities if $p\equiv 1~\text{mod}~4$ and it implies $K=\mathbb{Q}(\sqrt{-p})$ when $p\equiv 3~\text{mod}~4$ . In the latter case the dimension is $h(-p)$ because by the theory of complex multiplication, the elliptic curves with complex multiplication by $\mathbb{Q}(\sqrt{-p})$ are all isogenous, $\text{Gal}(\overline{\mathbb{Q}}/\mathbb{Q})$ -conjugate and their cardinality is $h(-p)$ .

Then we can estimate $h(-p)$ using Dirichlet’s class number formula:

$$\begin{eqnarray}h(-p)=-\frac{1}{p}\mathop{\sum }_{m=1}^{p-1}m\left(\frac{m}{p}\right)\leqslant \frac{1}{p}\mathop{\sum }_{m=1}^{p-1}m=\frac{p-1}{2}\end{eqnarray}$$

where the first equality is [Reference DavenportDav80, p. 51, (19)] (we are assuming $p\equiv 3~\text{mod}~4$ and $p\neq 3$ ), and in the inequality we used that the Legendre symbol $\left(\frac{m}{p}\right)$ takes values in $\{\pm 1\}$ . This implies $g_{\text{ns}}(p)\leqslant p$ . Contradiction.◻

Proof. By [Reference BaranBar10, Theorem 7.2], we have that $g_{\text{ns}}(p)>p$ for $p\geqslant 19$ . Regarding $p=13$ , $17$ , they are both congruent to $1$ modulo $4$ , so in these cases $g_{\text{ns}}^{C}(p)=g_{0}^{C}(p^{2})=0$ and we can use Proposition 2.4 together with the isogeny over $\mathbb{Q}$ between $J_{\text{ns}}(p)$ and the new part of $J_{0}(p^{2})$ . The automorphisms of $X_{\text{ns}}(11)$ are explicitly computed in [Reference Dose, Fernández, González and SchoofDFGS14] and they are all defined over $\mathbb{Q}$ .◻

Proof. Since $J_{\text{ns}}(p)$ is isogenous over $\mathbb{Q}$ to the new part of $J_{0}(p^{2})$ , an automorphism $u$ of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ induces an element of $\text{End}_{\mathbb{Q}}(J_{0}(p^{2}))\otimes \mathbb{Q}\cong (\text{End}_{\mathbb{Q}}(J_{\text{ns}}^{H}(p))\otimes \mathbb{Q})\times (\text{End}_{\mathbb{Q}}(J_{\text{ns}}^{C}(p))\otimes \mathbb{Q})$ , as explained above. Since $J_{0}(p^{2})$ has good reduction outside $p$ , the reduction modulo $l$ map is injective on the endomorphism ring of $J_{0}(p^{2})$ for every prime number $l\neq p$ . Hence, by the Eichler–Shimura relations, $u$ commutes with every Hecke operator $T_{l}$ with $l\neq p$ , because $u$ is defined over $\mathbb{Q}$ . Moreover, the Hecke algebra acting on the new part is generated by the operators $T_{l}$ with $l\neq p$ , thus $u$ commutes with the whole Hecke algebra. Then, to prove the Proposition, we note that every factor of $J_{0}^{H}(p^{2})$ is without complex multiplication and so the Hecke algebra is its own commutant in $\text{End}(J_{0}^{H}(p^{2}))\otimes \mathbb{Q}$ (Proposition 2.2).◻

Proof. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ . Then, by the previous Proposition, $u$ and $w$ induce elements in the Hecke algebra of $J_{0}^{H}(p^{2})$ , which is a commutative algebra. Hence, if we define $v=uwu^{-1}w^{-1}$ we have that $v$ induces an invertible element of $\text{End}(J_{0}(p^{2}))\otimes \mathbb{Q}$ which acts trivially on $J_{0}^{H}(p^{2})$ and $J_{\text{ns}}^{H}(p)$ . Then we can use the same proof as for Theorem 2.5 to see that $v$ is trivial if $g_{\text{ns}}(p)>p$ .◻

Proof. As for Corollary 2.6, we have that $g_{\text{ns}}(p)>p$ for $p\geqslant 19$ . Regarding $p=13$ , $17$ , in these cases $g_{\text{ns}}^{C}(p)=g_{0}^{C}(p^{2})=0$ . Moreover, $g_{\text{ns}}(p)\geqslant 2$ , so the natural map from the automorphism group of $X_{\text{ns}}(p)$ to the automorphism group of $J_{\text{ns}}(p)$ is injective [Reference Baker and HasegawaBH03, Lemma 2.1], and we can use Proposition 2.7 together with the fact that the Hecke algebra is commutative. The automorphism group of $X_{\text{ns}}(11)$ is the Klein four group (see [Reference Dose, Fernández, González and SchoofDFGS14]).◻

Proof. Let $C$ be the set of cusps on $X_{H}$ . Note that an automorphism $u$ of $X_{H}$ preserving the cusps and the elliptic points, defines naturally an automorphism of the open Riemann surface $X_{H}(\mathbb{C})\smallsetminus (C\cup \unicode[STIX]{x1D70B}(E))$ . If $X_{H}$ has no elliptic points ( $E=\emptyset$ ), then the quotient ${\mathcal{H}}\rightarrow {\mathcal{H}}/\unicode[STIX]{x1D6E4}_{H}\cong X_{H}(\mathbb{C})\smallsetminus C$ is the universal cover of $X_{H}(\mathbb{C})\smallsetminus C$ . Then $u$ automatically lifts to an automorphism of ${\mathcal{H}}$ . If instead $E\neq \emptyset$ , to lift $u$ to ${\mathcal{H}}\smallsetminus E$ , we need the push-forward by $u\circ \unicode[STIX]{x1D70B}$ of the fundamental group of ${\mathcal{H}}\smallsetminus E$ to be contained in its push-forward by $\unicode[STIX]{x1D70B}$ . But this is true because $u$ actually extends to an automorphism of $X_{H}(\mathbb{C})$ preserving $C$ and $\unicode[STIX]{x1D70B}(E)$ . Hence $u$ lifts to an automorphism of ${\mathcal{H}}\smallsetminus E$ , which extends uniquely to ${\mathcal{H}}$ .◻

Proof. Since the genus of $X_{\text{ns}}(p)$ is at least 2, the group $B(X_{\text{ns}}(p))$ must be finite. This means that $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ is commensurable with $\text{SL}_{2}(\mathbb{Z})$ in the sense that $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))\cap \text{SL}_{2}(\mathbb{Z})$ has finite index in both $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ and $\text{SL}_{2}(\mathbb{Z})$ . Hence $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ is a group acting on lattices of $\mathbb{R}^{2}$ that are commensurable with $\mathbb{Z}\times \mathbb{Z}$ , with the action given by right multiplication of row vectors in the lattice. Lattices commensurable with $\mathbb{Z}\times \mathbb{Z}$ are the ones generated by two vectors $(a,b)$ , $(c,d)$ with $a,b,c,d\in \mathbb{Q}$ , and if we consider them up to multiplication by a scalar (i.e., homothety), they all have a basis of the form $\{(M,\frac{g}{h}),(0,1)\}$ with $M\in \mathbb{Q}^{+}$ , $g,h\in \mathbb{Z}$ , $\text{gcd}(g,h)=1$ and $0\leqslant g<h$ (see [Reference ConwayCon96] for more details).

Since $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)\subset \unicode[STIX]{x1D6E4}(p)=\left\{\unicode[STIX]{x1D6FE}\in \text{SL}_{2}(\mathbb{Z})\text{ s. t. }\unicode[STIX]{x1D6FE}\equiv \text{Id mod }p\right\}$ , the lattices fixed by $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ are a subset of the lattices fixed by $\unicode[STIX]{x1D6E4}(p)$ . The lattices fixed by $\unicode[STIX]{x1D6E4}(p)$ are those who belong to one of the following type (see [Reference LangLan02, Lemma 4.1])

• ∙ $M=\frac{1}{p}$ , $h=p$ , $g=0,\ldots ,p-1$ , which gives a lattice homothetic to $\langle (1,g),(0,p)\rangle$ ;

• ∙ $M=\frac{1}{p}$ , $h=1$ , $g=0$ , which gives a lattice homothetic to $\langle (1,0),(0,p)\rangle$ ;

• ∙ $M=p$ , $h=1$ , $g=0$ , which gives the lattice $\langle (p,0),(0,1)\rangle$ ;

• ∙ $M=1$ , $h=1$ , $g=0$ , which gives the lattice $\langle (1,0),(0,1)\rangle =\mathbb{Z}\times \mathbb{Z}$ .

Recall that $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ is the subgroup of $\text{SL}_{2}(\mathbb{Z})$ made up of the elements $\unicode[STIX]{x1D6FE}$ such that

$$\begin{eqnarray}\unicode[STIX]{x1D6FE}\equiv \left(\begin{array}{@{}cc@{}}x & \unicode[STIX]{x1D6FC}y\\ y & x\end{array}\right)~\text{mod}~p\end{eqnarray}$$

for some $(x,y)\in \mathbb{F}_{p}^{2}$ and a nonsquare element $\unicode[STIX]{x1D6FC}\in \mathbb{F}_{p}$ . Then, a straightforward computation modulo $p$ shows that $\unicode[STIX]{x1D6FE}$ fixes lattices of the first three types, only if $\unicode[STIX]{x1D6FE}\in \unicode[STIX]{x1D6E4}(p)$ , so that the only lattice up to homothety fixed by $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ is $\mathbb{Z}\times \mathbb{Z}$ .

The normalizer in $\text{SL}_{2}(\mathbb{R})$ of $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ must preserve, by definition, the set of lattices that are fixed by $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ . Thus, also $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ fixes $\mathbb{Z}\times \mathbb{Z}$ . The stabilizer of $\mathbb{Z}\times \mathbb{Z}$ is $\text{SL}_{2}(\mathbb{Z})$ , implying $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))\subset \text{SL}_{2}(\mathbb{Z})$ . Therefore, $\text{Norm}(\unicode[STIX]{x1D6E4}_{\text{ns}}(p))$ is made up of the elements of $\text{SL}_{2}(\mathbb{Z})$ which reduce modulo $p$ to an element in the normalizer of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ , which proves the Proposition.◻

Proof. Let $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ be the subgroup of $\text{SL}_{2}(\mathbb{Z})$ made up of the elements which reduce modulo $p$ to an element in a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ , so that $X_{\text{ns}}(p)(\mathbb{C})\cong {\mathcal{H}}\cup \mathbb{Q}\cup \{\infty \}/\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ . Thus, $X_{\text{ns}}(p)$ has elliptic points if and only if $\unicode[STIX]{x1D6E4}_{\text{ns}}(p)$ contains an element with characteristic polynomial equal to $x^{2}+1$ or $x^{2}+x+1$ . Recall that the elements of a nonsplit Cartan subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ (not belonging to the center) have irreducible characteristic polynomials. If $p\equiv 1~\text{mod}~4$ and $p\equiv 1~\text{mod}~3$ , the two polynomials above have both roots over $\mathbb{F}_{p}$ . Hence, when $p\equiv 1~\text{mod}~12$ there are no elliptic points on $X_{\text{ns}}(p)$ . The Corollary then follows from the two previous Propositions.◻

Proof. (Following [Reference Kenku and MomoseKM88, p. 64, Lemma 2.6]) Since $J_{\text{ns}}(p)$ has good reduction outside $p$ , the reduction modulo $l$ map is injective on the endomorphism ring of $J_{\text{ns}}(p)$ . Hence, the Lemma follows from the Eichler–Shimura relations and the fact that $K(p)$ is a quadratic field.◻

Proof. The equality must hold modulo any prime ideal $\mathfrak{l}$ of $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ over $l$ because of the Eichler–Shimura relations. Moreover, since $l\neq p$ , the modular curve $X_{\text{ns}}(p)$ and its reduction modulo $l$ have the same number of cusps. Therefore, reduction modulo $\mathfrak{l}$ is injective on the set of cusps of $X_{\text{ns}}(p)$ , and the equality holds also over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ .◻

Proof. By the previous Proposition, for every cusp $C$ , we have $T_{l}C=C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}$ . Thus, we can choose a cusp $C^{\prime }$ such that the supports of $T_{l}C$ and $T_{l}C^{\prime }$ are completely disjoint. For $D_{l}$ to be the zero divisor, we must have

$$\begin{eqnarray}u^{\unicode[STIX]{x1D70E}_{l}}C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}+T_{l}uC^{\prime }=T_{l}uC+u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}.\end{eqnarray}$$

Since we chose $C^{\prime }$ such that the supports of $T_{l}C$ and $T_{l}C^{\prime }$ are completely disjoint, we have

(4.2) $$\begin{eqnarray}T_{l}uC=u^{\unicode[STIX]{x1D70E}_{l}}C^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}C^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}.\end{eqnarray}$$

If $uC$ is not a cusp, it corresponds to an elliptic curve $E$ defined over $\mathbb{Q}(uC)\subset \mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ .

Let $A$ and $B$ be two different cyclic groups of order $l$ in $E$ . Then the natural map $E/A\rightarrow E\rightarrow E/B$ has a cyclic kernel of order $l^{2}$ . Therefore, recalling the interpretation of the action of $T_{l}$ on noncuspidal points, we have that equation (4.2) can hold only if $E$ admits an endomorphism of degree $l^{2}$ with cyclic kernel. An elliptic curve without complex multiplication cannot have such an endomorphism, hence we obtain the Lemma as a consequence of the following Proposition.◻

Proof. Let $P$ be a point of $X_{\text{ns}}(p)$ associated to $E$ and defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ . There are three cases: $p$ is inert, $p$ splits or $p$ ramifies in $K$ .

If $p$ is inert in $K$ , we consider the point $Q$ of $X_{\text{ns}}^{+}(p)$ given by $\{P,wP\}$ . This point $Q$ is defined over $L\subset \mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ and lifts to points of $X_{\text{ns}}(p)$ defined over $LK$ (and no smaller fields) as explained in [Reference SerreSer89, p. 194–195]. But this is impossible since $P,wP$ are defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ where $p$ ramifies completely.

If $p$ splits or ramifies in $K$ , this means that the image of the $\text{Gal}(\overline{\mathbb{Q}}/K(\unicode[STIX]{x1D701}_{p}))$ -representation modulo $p$ attached to $E$ is contained in respectively a split Cartan or a Borel subgroup of $\text{GL}_{2}(\mathbb{F}_{p})$ (see loc. cit.). But since $P$ is defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ , the image of such representation is also contained in a nonsplit Cartan subgroup. Therefore, in both cases, it is contained in the subgroup of scalar matrixes. Then, the properties of the Weil pairing imply that $\text{Gal}(\overline{\mathbb{Q}}/K(\unicode[STIX]{x1D701}_{p}))$ acts on the $p$ -torsion of $E$ as a subgroup of $\{\pm 1\}$ . After choosing a suitable Weierstrass equation for $E$ , this means that the $x$ -coordinates of the $p$ -torsion points of $E$ are in $K(\unicode[STIX]{x1D701}_{p})$ . The theory of complex multiplication tells us that the ray class field $K_{(p)}$ modulo $(p)$ of $K$ is generated over $K$ by the $j$ -invariant of $E$ and by some rational function of the $x$ -coordinates of the $p$ -torsion points of $E$ [Reference SilvermanSil94, p. 135, Theorem 5.6]. Thus we obtain $K_{(p)}\subset K(\unicode[STIX]{x1D701}_{p})$ . It is always true that $K(\unicode[STIX]{x1D701}_{p})\subset K_{(p)}$ , so the opposite inclusion holds if and only if $[K_{(p)}:K]\leqslant [K(\unicode[STIX]{x1D701}_{p}):K]$ . We have

$$\begin{eqnarray}[K_{(p)}:K]=h_{K}\frac{\#(O_{K}/pO_{K})^{\ast }}{w_{K}}\end{eqnarray}$$

where $h_{K}$ is the class number of $K$ , $O_{K}$ is the ring of integers of $K$ and $w_{K}$ is the number of roots of unity in $K$ taken modulo $(p)$ . The group $(O_{K}/pO_{K})^{\ast }$ is isomorphic to $\mathbb{F}_{p^{2}}^{\ast }$ , $\mathbb{F}_{p}^{\ast }\times \mathbb{F}_{p}^{\ast }$ or $\mathbb{F}_{p}^{\ast }\times \langle a\rangle$ with $a$ an element of order $p$ . Thus to have the inequality $[K_{(p)}:K]\leqslant [K(\unicode[STIX]{x1D701}_{p}):K]$ we must have

$$\begin{eqnarray}\frac{(p-1)^{2}}{w_{K}}\leqslant (p-1)\end{eqnarray}$$

and hence $p\leqslant w_{K}+1$ . Since the roots of unity in an imaginary quadratic field are at most $6$ , we get $p\leqslant 7$ .◻

Proof. Since $u$ is defined over $K(p)$ and does not preserve the cusps, we can apply Lemma 4.3 to obtain, for every prime $l\neq p$ , a nonzero divisor $D_{l}$ on $X_{\text{ns}}(p)$ , of degree zero, defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ , which is the difference of two effective divisors of degree at most $2(l+1)$ . Moreover, we have $D_{l}=(u^{\unicode[STIX]{x1D70E}_{l}}T_{l}-T_{l}u)(C-C^{\prime })$ for two cusps $C$ , $C^{\prime }$ of $X_{\text{ns}}(p)$ . Hence, by Lemma 4.1 we deduce that $D_{l}$ must be zero in $J_{\text{ns}}(p)$ when $p\geqslant 11$ , which is to say that $D_{l}$ is a principal divisor. This gives us the existence of a nonconstant morphism $f$ from $X_{\text{ns}}(p)$ to $\mathbb{P}^{1}$ , defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ and with degree at most $2(l+1)$ .

Now we apply $w$ to $D_{l}$ . If $D_{l}$ were invariant under the action of $w$ , we would have

(4.3) $$\begin{eqnarray}wT_{l}uC+wu^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{\unicode[STIX]{x1D70E}_{l}}+l\cdot wu^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{{\unicode[STIX]{x1D70E}_{l}}^{-1}}=T_{l}uC+u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{\unicode[STIX]{x1D70E}_{l}}+l\cdot u^{\unicode[STIX]{x1D70E}_{l}}{C^{\prime }}^{{\unicode[STIX]{x1D70E}_{l}}^{-1}},\end{eqnarray}$$

where we used the computation in the proof of Lemma 4.3. In that proof, we also showed that the divisor $T_{l}uC$ is the sum of $l+1$ different points. Hence since $l\geqslant 2$ , to satisfy equation (4.3), the supports of $T_{l}uC$ and $wT_{l}uC$ must have a nonempty intersection. This means that the support of $T_{l}uC$ has either two different points exchanged by $w$ or a point fixed by $w$ . The first case is impossible because the modular automorphism $w$ preserves the isomorphism class over $\mathbb{C}$ of the elliptic curve associated to noncuspidal points, and we can apply the same argument at the end of the proof of Lemma 4.3. The second case is also impossible by the previous Proposition, because the points fixed by $w$ are all associated with the elliptic curve with $j$ -invariant equal to $1728$ (see [Reference BaranBar10, Proposition 7.10]), which is an elliptic curve with complex multiplication.

Thus, for $l\geqslant 2$ , we have $wD_{l}\neq D_{l}$ which implies $f\circ w\neq f$ . The degree of $f$ is $2(l+1)$ , so by choosing $l=2$ we get the Corollary.◻

Proof. [Reference Baker and HasegawaBH03, Lemma 3.5] Consider the rational function $g=f\circ w-f$ on $X$ . If $g$ is constant, then the fixed points of $w$ must be poles of $f$ , implying $r\leqslant k$ . If $g$ is not constant then the degree of $g$ is at most $2k-h$ , where $h$ is the number of poles that are also points fixed by $w$ . On the other hand, the other $r-h$ points fixed by $w$ are zeros of $g$ , hence there are at most $\text{deg }g$ of them. So we have $r-h\leqslant 2k-h$ which implies $r\leqslant 2k$ .◻

Proof. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ not preserving the cusps. By Corollary 2.6 we have that $u$ is defined over $K(p)$ . Then Corollary 4.5 tells us that there exists a nonconstant morphism from $X_{\text{ns}}(p)$ to $\mathbb{P}^{1}$ , defined over $\mathbb{Q}(\unicode[STIX]{x1D701}_{p})$ , with degree less or equal to $6$ and such that $f\circ w\neq f$ . Then by the previous Lemma we have

$$\begin{eqnarray}\#\{\text{fixed points of }w\}\leqslant 12.\end{eqnarray}$$

Recall that the number of fixed points of $w$ is $\frac{p-1}{2}$ if $p\equiv 1~\text{mod}~4$ and it is $\frac{p+1}{2}$ if $p\equiv 3~\text{mod}~4$ (see [Reference BaranBar10, Proposition 7.10]). This implies $p\leqslant 23$ and proves the first part of the Theorem. The second part is now a consequence of Corollary 3.3.◻

Proof. Let $u$ be an automorphism of $X_{\text{ns}}(p)$ defined over $\mathbb{Q}$ . By Corollary 2.9 we have that $u$ commutes with $w$ and it induces an automorphism of $X_{\text{ns}}^{+}(p)$ defined over $\mathbb{Q}$ . Then it is enough to prove that $u$ preserves the rational CM points of $X_{\text{ns}}^{+}(p)$ only if $u$ is modular. Let $\unicode[STIX]{x1D70B}:X_{\text{ns}}(p)\rightarrow X_{\text{ns}}^{+}(p)$ be the degree-2 modular morphism given by the inclusion of a nonsplit Cartan subgroup in its normalizer, and let $Q$ be a rational CM point of $X_{\text{ns}}^{+}(p)$ . Note that the two points in $\unicode[STIX]{x1D70B}^{-1}(Q)$ are defined over the CM field of the elliptic curve associated to $Q$ (see [Reference SerreSer89, p. 194–195]). Hence $uQ$ cannot be a rational CM point different from $Q$ because the two points in $\unicode[STIX]{x1D70B}^{-1}(uQ)$ are defined over the same field as the points in $\unicode[STIX]{x1D70B}^{-1}(Q)$ . Now we observe that the elliptic points of $X_{\text{ns}}(p)$ are the inverse images by $\unicode[STIX]{x1D70B}$ of the rational CM points associated to the elliptic curves with $j$ -invariant equal to 0 or 1728 (see [Reference BaranBar10, Proposition 7.10]). Thus, if $u$ preserves the rational CM points of $X_{\text{ns}}^{+}(p)$ , then it preserves the elliptic points of $X_{\text{ns}}(p)$ . Furthermore, $u$ also preserves the cusps by Theorem 5.2, hence it is modular by Corollary 3.3.◻