2 Preliminary calculations

For simplicity, we write $c_r$ for the coefficients $c_r(m,N,k)$ . Let $d=\dim S_k(\Gamma _0(N))$ and $\lambda _1, \ldots , \lambda _d$ denote the eigenvalues of $T^{\prime }_m(N,k)$ . Observe that $(-1)^r c_r$ is just the rth elementary symmetric polynomial of these eigenvalues:

$$ \begin{align*} c_0=1,\quad -c_1 = \sum_{1\leq i\leq d} \lambda_i \,, \quad c_2 = \sum_{1\leq i < j \leq d} \lambda_i \lambda_j \,, \quad -c_3 = \sum_{1\leq i<j<\ell\leq d} \lambda_i \lambda_j \lambda_\ell\,, \quad \ldots \end{align*} $$

We also write $p_r$ for the sum of rth powers of these eigenvalues:

(1) $$ \begin{align} p_r:=\sum_{i=1}^{d} \lambda_i^r. \end{align} $$

Then the Girard–Newton identities yield the following relation between the $c_r$ and the $p_r$ .

Lemma 2.1 [Reference Tignol13, p. 38]

Let $c_r$ and $p_r$ be defined as above. Then for $r\ge 1$ ,

$$ \begin{align*} c_r = \frac{-1}{r} \sum_{j=1}^r c_{r-j} p_j. \end{align*} $$

We also give estimates on the traces of Hecke operators. These estimates will be needed shortly when we express the $p_r$ in terms of traces of certain Hecke operators. In a previous paper [Reference Ross and Xue11], we proved the following result by analyzing the various terms of the Eichler–Selberg trace formula.

Lemma 2.2 [Reference Ross and Xue11, Lemmas 4.1 and 4.2]

Fix an integer $m \ge 1$ . Then for N coprime to m and $k \geq 2$ even,

$$ \begin{align*} \operatorname{\mathrm{Tr}} T^{\prime}_m(N,k) = \begin{cases} \frac{1}{\sqrt{m}}\frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}), & \text{if }m\text{ is a perfect square}, \\ O(N^\varepsilon), & \text{if }m\text{ is not a perfect square}. \end{cases} \end{align*} $$

To gauge the growth of the terms in this formula, note that $\psi (N) \geq N$ and $\psi (N) = O(N^{1+\varepsilon })$ [Reference Hardy and Wright7, Sections 18.1 and 22.13].

3 When $m$ is a perfect square

In this section, we consider the case when m is a perfect square. We then have the following estimates on the $p_j$ (1).

Lemma 3.1 Fix an integer $r \ge 1$ and a perfect square $m \ge 1$ . Then for N coprime to m and $k \geq 2$ even,

$$ \begin{align*} p_1 &= \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}), \qquad \text{and} \\ p_{j} &= O(kN^{1+\varepsilon}), \qquad \text{for all }1 \leq j \leq r. \end{align*} $$

Proof The first claim follows immediately from Lemma 2.2.

For the second claim, note from Lemma 2.2 and the fact that $\psi (N)=O(N^{1+\epsilon })$ ,

$$ \begin{align*} d := \dim S_k(\Gamma_0(N)) = \operatorname{\mathrm{Tr}} T^{\prime}_1 = O(kN^{1+\varepsilon}). \end{align*} $$

Then utilizing Deligne’s bound $\left \lvert \lambda _i \right \lvert \leq \sigma _0(m) = \sum _{d\mid m} 1$ , we obtain

$$ \begin{align*}\left\lvert p_j \right\lvert = \left\lvert \sum_{i=1}^d \lambda_i^j \right\lvert \leq \sum_{i=1}^d \sigma_0(m)^j = O(kN^{1+\varepsilon}), \end{align*} $$

as desired.

For m and r fixed, we now determine the asymptotic behavior of $c_r(m,N,k)$ as $N+k \to \infty $ . Note $c_r(m,N,k)$ is not technically defined for $N,k$ such that $\dim S_k(\Gamma _0(N)) < r$ . However, there are only finitely many such pairs $(N,k)$ [Reference Ross10, Theorem 1.1], so it is well-defined here to ask about $c_r(m,N,k)$ as $N+k\to \infty $ .

Theorem 1.1 Fix an integer $r \ge 0$ and a perfect square $m \ge 1$ . Then for N coprime to m and $k \geq 2$ even,

$$ \begin{align*} c_r(m,N,k) = \frac{(-1)^r}{r!} \left( \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi(N)\right)^r + O(k^{r-1}N^{r-1/2+\varepsilon}). \end{align*} $$

Proof We proceed by strong induction on r. The base case of $r=0$ is immediate since $c_0 = 1$ .

For $r \geq 1$ , we have by Lemma 2.1 that

(2) $$ \begin{align} c_{r} = \frac{-1}{r} \sum_{j=1}^{r} c_{r-j} p_j = \frac{-1}{r} \Bigg[ c_{r-1} p_1 + \sum_{j=2}^{r} c_{r-j} p_j \Bigg]. \end{align} $$

Then by the induction hypothesis,

$$ \begin{align*} c_{r-1} &= \frac{(-1)^{r-1}}{(r-1)!} \left( \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi(N)\right)^{r-1} + O(k^{r-2}N^{r-3/2+\varepsilon}), \\ c_{r-j} &= O(k^{r-2} N^{r-2+\varepsilon}), \qquad\qquad\qquad \text{for }2\le j \le r, \end{align*} $$

and by Lemma 3.1,

$$ \begin{align*} p_1 &= \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}), \\ p_j &= O(kN^{1+\varepsilon}), \qquad\qquad\qquad \text{for }2\le j \le r. \end{align*} $$

Applying these estimates to (2), we obtain

$$ \begin{align*} c_{r} &= \frac{-1}{r} \Bigg[ c_{r-1} p_1 + \sum_{j=2}^{r} c_{r-j} p_j \Bigg] \\&= \frac{-1}{r} \Bigg[ \left(\frac{(-1)^{r-1}}{(r-1)!} \left(\frac{1}{\sqrt{m}} \frac{k-1}{12} \psi(N)\right)^{r-1} + O(k^{r-2} N^{r-3/2+\varepsilon}) \right) \\& \quad\qquad \times \left( \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}) \right) + \sum_{j=2}^{r} O(k^{r-2} N^{r-3/2+\varepsilon}) \cdot O(k N^{1+\varepsilon}) \Bigg] \\&= \frac{-1}{r} \Bigg[ \frac{(-1)^{r-1}}{(r-1)!} \left(\frac{1}{\sqrt m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}) \Bigg] \\&= \frac{(-1)^r}{r!} \left(\frac{1}{\sqrt m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}). \end{align*} $$

This completes the proof.

Theorem 1.1 allows us in particular to determine the sign of $c_r(m,N,k)$ for all but finitely many pairs $(N,k)$ .

Proof Since $\psi (N) \ge N$ , we can write the asymptotic formula from Theorem 1.1 as

$$ \begin{align*} c_r(m,N,k) &= \frac{(-1)^r}{r! \sqrt{m}^r} \left(\frac{k-1}{12} \psi(N)\right)^r \left[1 + O\left(k^{r-1}N^{r-1/2+\varepsilon}\right) \left(\frac{k-1}{12} \psi(N)\right)^{-r} \right] \\ &= \frac{(-1)^r}{r! \sqrt{m}^r} \left(\frac{k-1}{12} \psi(N)\right)^r \left[ 1 + O\left(k^{-1}N^{-1/2+\varepsilon}\right) \right]. \end{align*} $$

Then since the $O\left (k^{-1}N^{-1/2+\varepsilon }\right )$ term tends to $0$ as $N \to \infty $ or $k \to \infty $ , it will have magnitude less than $1$ for all but finitely many pairs $(N,k)$ . This then yields the desired result.

4 When $m$ is not a perfect square

In this section, we consider the remaining case when m is not a perfect square. First, we have the following estimates on the $p_j$ (1).

Lemma 4.1 Fix an integer $r \ge 0$ and a non-square $m \ge 1$ . Then

$$ \begin{align*} p_1 &= -c_1 = O(N^\varepsilon), \\ p_2 &= \frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}), \\ p_{3} &= O(N^\varepsilon), \\ p_{j} &= O(kN^{1+\varepsilon}), \qquad \text{for all }1 \leq j \leq r. \end{align*} $$

Proof The first claim follows immediately from Lemma 2.2.

For the second claim, observe that $p_2 = \operatorname {\mathrm {Tr}} {T_m'}^{2}$ . Then by the Hecke operator composition formula [Reference Cohen and Strömberg6, Theorem 10.2.9] and Lemma 2.2,

$$ \begin{align*} p_2 = \operatorname{\mathrm{Tr}} {T_m'}^2 = \sum_{d \mid m} \operatorname{\mathrm{Tr}} T_{m^2/d^2}' &= \sum_{d \mid m} \frac{d}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}) \\ &= \frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}). \end{align*} $$

For the third claim, we similarly have by the Hecke operator composition formula and Lemma 2.2,

$$ \begin{align*} p_3 = \operatorname{\mathrm{Tr}} {T_m'}^3 = \operatorname{\mathrm{Tr}} \sum_{d \mid m} T_{m^2/d^2}' T_m' &= \sum_{d \mid m} \ \sum_{\delta \mid (m^2/d^2, m)} \operatorname{\mathrm{Tr}} T^{\prime}_{m^3/d^2\delta^2} \\ &= \sum_{d \mid m} \ \sum_{\delta \mid (m^2/d^2, m)} O(N^\varepsilon) \\ &= O(N^{\varepsilon}). \end{align*} $$

Finally, the fourth claim follows from an identical argument as in Lemma 3.1.

For m and r fixed, we now determine the asymptotic behavior of $c_r(m,N,k)$ as $N+k \to \infty $ .

Theorem 1.2 Fix an integer $r \ge 0$ and a non-square $m \ge 1$ . Then for N coprime to m and $k \geq 2$ even,

$$ \begin{align*} c_{2r} &= \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^r + O(k^{r-1} N^{r-1/2+\varepsilon}) \qquad \text{and} \\ c_{2r+1} &= c_1 \cdot \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12}\psi(N)\right)^r + O(k^{r-1} N^{r-1/2+\varepsilon}). \end{align*} $$

Proof We proceed by strong induction on r. The base case of $r=0$ is immediate since $c_0 = 1$ and $c_1 = c_1$ .

Then for $r\ge 1$ , we have from Lemma 2.1 that

(3) $$ \begin{align} c_{2r} &= \frac{-1}{2r} \sum_{j=1}^{2r} c_{2r-j} p_j = \frac{-1}{2r} \Bigg[ c_{2r-1} p_1 + c_{2r-2} p_2 + \sum_{j=3}^{2r} c_{2r-j} p_j \Bigg]. \end{align} $$

Then by the induction hypotheses,

$$ \begin{align*} c_{2r-1} &= O(k^{r-1} N^{r-1+\varepsilon}), \\ c_{2r-2} &= \frac{(-1)^{r-1}}{(2r-2)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r-1} + O(k^{r-2} N^{r-3/2+\varepsilon}), \\ c_{2r-j} &= O(k^{r-2} N^{r-2+\varepsilon}), \qquad\qquad \text{for }3\le j \le 2r, \end{align*} $$

and by Lemma 4.1,

$$ \begin{align*} p_1 &= O(N^\varepsilon), \\ p_2 &= \frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}), \\ p_j &= O(kN^{1+\varepsilon}), \qquad\qquad \text{for }3\le j \le 2r. \end{align*} $$

Applying these estimates to (3), we obtain

$$ \begin{align*} c_{2r} &= \frac{-1}{2r} \Bigg[ c_{2r-1} p_1 + c_{2r-2} p_2 + \sum_{j=3}^{2r} c_{2r-j} p_j \Bigg] \\&= \frac{-1}{2r} \Bigg[ O(k^{r-1} N^{r-1+\varepsilon}) \cdot O(N^\varepsilon) \\&\qquad\ \ + \left(\frac{(-1)^{r-1}}{(2r-2)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r-1} + O(k^{r-2} N^{r-3/2+\varepsilon}) \right) \\&\qquad\ \ \times \left( \frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}) \right) + \sum_{j=3}^{2r} O(k^{r-2} N^{r-2+\varepsilon}) \cdot O(k N^{1+\varepsilon}) \Bigg] \\&= \frac{-1}{2r} \Bigg[ O(k^{r-1}N^{r-1+\varepsilon}) + \frac{(-1)^{r-1}}{(2r-2)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} \\&\qquad\ \ + O(k^{r-1} N^{r-1/2+\varepsilon}) + O(k^{r-1} N^{r-1+\varepsilon}) \Bigg] \\&= \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}), \end{align*} $$

verifying the first claim of the inductive step.

For the second claim of the inductive step, we similarly have from Lemma 2.1 that

(4) $$ \begin{align} c_{2r+1} &= \frac{-1}{2r+1} \sum_{j=1}^{2r+1} c_{2r+1-j} p_j = \frac{-1}{2r+1} \Bigg[ c_{2r} p_1 + c_{2r-1} p_2 + c_{2r-2} p_3 + \sum_{j=4}^{2r+1} c_{2r+1-j} p_j \Bigg]. \end{align} $$

Then by the induction hypotheses and the proof for $c_{2r}$ ,

$$ \begin{align*} c_{2r} &= \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}), \\ c_{2r-1} &= c_1 \cdot \frac{(-1)^{r-1}}{(2r-2)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12}\psi(N)\right)^{r-1} + O(k^{r-2} N^{r-3/2+\varepsilon}), \\ c_{2r-2} &= O(k^{r-1} N^{r-1+\varepsilon}), \\ c_{2r+1-j} &= O(k^{r-2} N^{r-2+\varepsilon}), \qquad\qquad \text{for }4\le j \le 2r+1, \end{align*} $$

and by Lemma 4.1,

$$ \begin{align*} p_1 &= -c_1, \\ p_2 &= \frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon}), \\ p_3 &= O(N^\varepsilon), \\ p_j &= O(kN^{1+\varepsilon}), \qquad\qquad \text{for }4\le j \le 2r+1. \end{align*} $$

Applying these estimates to (4), we obtain

$$ \begin{align*} c_{2r+1} &= \frac{-1}{2r+1} \Bigg[ c_{2r} p_1 + c_{2r-1} p_2 + c_{2r-2} p_3 + \sum_{j=4}^{2r+1} c_{2r+1-j} p_j \Bigg] \\&= \frac{-1}{2r+1} \Bigg[\left(\frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}) \right) \cdot (-c_1) \\&\qquad\qquad + \left(c_1 \cdot \frac{(-1)^{r-1}}{(2r-2)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12}\psi(N)\right)^{r-1} + O(k^{r-2} N^{r-3/2+\varepsilon})\right) \\&\qquad\qquad \times \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N) + O(N^{1/2+\varepsilon})\right) \\&\qquad\qquad + O(k^{r-1} N^{r-1+\varepsilon}) \cdot O(N^\varepsilon) + \sum_{j=4}^{2r+1} O(k^{r-2} N^{r-2+\varepsilon}) \cdot O(kN^{1+\varepsilon}) \Bigg] \\&= \frac{-1}{2r+1} \Bigg[ -c_1 \cdot \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}) \\&\qquad\qquad + c_1 \cdot \frac{(-1)^{r-1}}{(2r-2)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12}\psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}) \\&\qquad\qquad + O(k^{r-1} N^{r-1+\varepsilon}) + O(k^{r-1} N^{r-1+\varepsilon}) \Bigg]\\&= c_1 \cdot \frac{1}{2r+1} \left(\frac{(-1)^r}{(2r)!!} - \frac{(-1)^{r-1}}{(2r-2)!!} \right) \cdot \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}) \\&= c_1 \cdot \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi(N)\right)^{r} + O(k^{r-1} N^{r-1/2+\varepsilon}), \end{align*} $$

verifying the second claim of the inductive step.

This completes the proof.

Theorem 1.2 allows us in particular to determine the sign of the even-indexed coefficients for all but finitely many pairs $(N,k)$ . The following corollary can be shown using an identical argument as in Corollary 3.2.

The behavior of the odd-indexed coefficients, on the other hand, is determined by the behavior of the trace.

Proof Since $\psi (N) \ge N$ , we can write the asymptotic formula from Theorem 1.2 as

$$ \begin{align*} c_{2r+1} &= \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m}\right)^r \left( \frac{k-1}{12}\psi(N)\right)^r \left[c_1 + O\left(k^{r-1} N^{r-1/2+\varepsilon}\right) \left( \frac{k-1}{12}\psi(N)\right)^{-r} \right] \\ &= \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m}\right)^r \left( \frac{k-1}{12}\psi(N)\right)^r \left[c_1 + O\left(k^{-1} N^{-1/2+\varepsilon}\right) \right]. \end{align*} $$

Then observe that since $\operatorname {\mathrm {Tr}} T_m \in {\mathbb {Z}}$ and $c_1 = -\operatorname {\mathrm {Tr}} T^{\prime }_m = - m^{-(k-1)/2} \operatorname {\mathrm {Tr}} T_m \neq 0$ , we must have $\left \lvert c_1 \right \lvert = \left \lvert \operatorname {\mathrm {Tr}} T^{\prime }_m \right \lvert \geq m^{-(k-1)/2}$ . And because the $O\left (k^{-1}N^{-1/2+\varepsilon }\right )$ term tends to $0$ as $N \to \infty $ , it will have magnitude less than $m^{-(k-1)/2}$ for all but finitely many N. This yields the desired result.

We note that the condition $\operatorname {\mathrm {Tr}} T^{\prime }_m(N,k) \neq 0$ here is not overly restrictive. Rouse [Reference Rouse12, Theorem 1.6] showed that there are only finitely many k for which we could possibly have $\operatorname {\mathrm {Tr}} T^{\prime }_m(N,k) = 0$ for some N. And even for these finitely many remaining k, he showed in [Reference Rouse12, Theorem 1.7] that $\operatorname {\mathrm {Tr}} T^{\prime }_m(N,k) \neq 0$ for $100\%$ of N. He further conjectured in [Reference Rouse12, Conjecture 1.5] that $\operatorname {\mathrm {Tr}} T^{\prime }_m(N,k) \neq 0$ for all $N \ge 1$ and $k=12$ or $\geq 16$ .

We also note that there are various ways one could try to improve this result to where N and k both vary. The only reason we fixed k was to guarantee that $c_1 = -\operatorname {\mathrm {Tr}} T^{\prime }_m(N,k)$ was bounded away from $0$ . If we relax the condition of k being fixed to just that $k \leq (1-\delta )\log _m(N)$ for some $\delta> 0$ , then we have the same result for all but finitely many pairs $(N,k)$ . One could also try to bound $\operatorname {\mathrm {Tr}} T^{\prime }_m(N,k)$ away from $0$ using some sort of vertical Atkin–Serre type result for the trace (e.g., along the lines of [Reference Kim8, Theorem 2.2]).

5 Extending to the new subspace

All of our results extend to the Hecke polynomial over the new subspace. Let $T^{\prime \,{\text {new}}}_m(N,k)$ denote the restriction of $T^{\prime }_m(N,k)$ to the new subspace $S_k^{\text {new}}(\Gamma _0(N))$ . Let $c^{\text {new}}_{r}(m,N,k)$ denote the rth coefficient of the characteristic polynomial $T^{\prime \,{\text {new}}}_{m}(N,k)(x)$ as follows:

$$ \begin{align*} T^{\prime \,{\text{new}}}_{m}(N,k)(x)=\sum_{r=0}^{d^{\text{new}}} c^{\text{new}}_r(m,N,k) x^{d^{\text{new}}-r}. \end{align*} $$

Here, $d^{\text {new}}=\dim S_k^{\text {new}}(\Gamma _0(N))$ . Just like before, we can then determine the asymptotic behavior of $c_r^{\text {new}}(m,N,k)$ as $N+k \to \infty $ . Note $c_r^{\text {new}}(m,N,k)$ is not technically defined for $N,k$ such that $\dim S_k^{\text {new}}(\Gamma _0(N)) < r$ . However, there are only finitely many such pairs $(N,k)$ [Reference Ross10, Theorem 1.3], so it is perfectly well-defined here to ask about $c_r^{\text {new}}(m,N,k)$ as $N+k\to \infty $ .

Let

$$ \begin{align*} \psi^{\text{new}}(N) := \prod_{p^r \parallel N} \begin{cases} p\left(1-\frac1p\right), & \text{if } r=1, \\ p^2\left(1-\frac1p - \frac{1}{p^2}\right), & \text{if } r=2, \\ p^r\left(1-\frac1p-\frac{1}{p^2}+\frac{1}{p^3}\right), & \text{if } r\geq3, \end{cases} \end{align*} $$

and note that $\psi ^{\text {new}}(N) \leq N$ and $\psi ^{\text {new}}(N) = \Omega (N^{1-\varepsilon })$ [Reference Hardy and Wright7, Sections 18.1 and 22.13]. In [Reference Cason, Jim, Medlock, Ross, Vilardi and Xue1, Lemmas 4.2 and 4.3], Cason et al. showed that for fixed m,

$$ \begin{align*} \operatorname{\mathrm{Tr}} T^{\prime \,{\text{new}}}_m(N,k) = \begin{cases} \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi^{\text{new}}(N) + O(N^{1/2}), & \text{if }m\text{ is a perfect square}, \\ O(N^\varepsilon), & \text{if }m\text{ is not a perfect square}. \end{cases} \end{align*} $$

The following two theorems then follow by an identical argument as in Theorems 1.1 and 1.2. The details are omitted.

Theorem 5.1 Fix an integer $r \ge 0$ and a perfect square $m \ge 1$ . Then for N coprime to m and $k \geq 2$ even,

$$ \begin{align*} c^{\mathrm{new}}_r(m,N,k) = \frac{(-1)^r}{r!} \left( \frac{1}{\sqrt{m}} \frac{k-1}{12} \psi^{\mathrm{new}}(N)\right)^r + O(k^{r-1}N^{r-1/2}). \end{align*} $$

Theorem 5.2 Fix an integer $r \ge 0$ and a non-square $m \ge 1$ . Then

$$ \begin{align*} c^{\mathrm{new}}_{2r}(m,N,k) &= \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12} \psi^{\mathrm{new}}(N)\right)^r + O(k^{r-1} N^{r-1/2}) \qquad \text{and} \\ c^{\mathrm{new}}_{2r+1}(m,N,k) &= c^{\mathrm{new}}_1(m,N,k) \cdot \frac{(-1)^r}{(2r)!!} \left(\frac{\sigma_1(m)}{m} \frac{k-1}{12}\psi^{\mathrm{new}}(N)\right)^r + O(k^{r-1} N^{r-1/2+\varepsilon}). \end{align*} $$

Then just like in Corollaries 3.2, 4.2, and 4.3, this tells us the sign of the $c^{\text {new}}_r$ in all but finitely many cases.

6 Sign patterns for more general polynomials

In response to our previous paper showing that $c_2$ tends to be negative [Reference Ross and Xue11], Kimball Martin suggested to us that $c_2$ might display a similar bias more generally for polynomials with totally real roots. In fact, the sign tendencies for all the coefficients given in Corollaries 4.2 and 4.3 hold more generally for a wide class of polynomials with totally real roots. Essentially the only two conditions we need to impose are that the roots are distributed over an interval $[-A,A]$ in a roughly symmetric way about the origin, and that the roots are not all clustered at the origin.

More precisely, for $A>0$ and r fixed, consider a sequence of polynomials $f_n$ with totally real roots lying in the interval $[-A,A]$ . Let $d_n$ denote the degree of $f_n$ , and let the $c_j(n)$ and $p_j(n)$ be defined as above in Section 2. We assume that $p_1(n) = o(d_n^{1/3})$ and $p_3(n) = o(d_n)$ (which will occur if the roots $x_1, \ldots , x_{d_n}$ of $f_n$ are distributed in a roughly symmetric way about the origin). Also note that $p_2(n) = \alpha _n^2 d_n$ , where $\alpha _n$ denotes the quadratic mean of the roots of $f_n$ . We assume that $\alpha _n$ is bounded away from $0$ (which will occur as long as the roots are not all clustered at the origin). Finally, observe that $\left \lvert p_j(n) \right \lvert \leq A^r d_n = O(d_n)$ for each $1 \leq j \leq r$ . These estimates

$$ \begin{align*} p_1(n) &= -c_1(n) = o(d_n^{1/3}), \\ p_2(n) &= \alpha_n^2 d_n, \\ p_{3}(n) &= o(d_n), \\ p_{j}(n) &= O(d_n) \qquad \text{for each }1 \leq j \leq r, \end{align*} $$

are essentially the content of Lemma 4.1 (except that Lemma 4.1 has much stronger error bounds, and where in that case, $\alpha _{N,k}$ tends to $\sqrt {\sigma _1(m)/m}$ as $N+k \to \infty $ [Reference Cason, Jim, Medlock, Ross and Xue2, Theorem 1.1]).

Then using an identical argument as in Theorem 1.2, one can show that

$$ \begin{align*} c_{2r}(n) &= \frac{(-1)^r}{(2r)!!} \left(\alpha_n^2 d_n\right)^r + o(d_n^{r-1/3}) \qquad \text{and} \\ c_{2r+1}(n) &= c_1(n) \frac{(-1)^r}{(2r)!!} \left(\alpha_n^2 d_n\right)^r + o(d_n^{r}). \end{align*} $$

In particular, this means that as $d_n \to \infty $ , the coefficients of $f_n$ will tend to the sign pattern

$$ \begin{align*} &+--++--++--++\cdots \qquad \text{if }c_1\text{ is bounded below }0,\\ &++--++--++--+\cdots \qquad \text{if }c_1\text{ is bounded above }0. \end{align*} $$

For example, browsing the polynomials with totally real roots of degree 10 in LMFDB, almost all of them follow this sign pattern; see [9] (such polynomials given by LMFDB are shifted so that their roots are roughly symmetric about the origin).

If the roots of a polynomial are perfectly symmetric about the origin, then we will have $c_1 = 0$ , and the sign pattern becomes

$$ \begin{align*} &+0-0+0-0+0-0+\cdots. \end{align*} $$

For example, the roots of the Chebyshev polynomials are distributed in $[-1,1]$ in a perfectly symmetric way, and their coefficients follow precisely this pattern.

We also note what happens when the roots of a polynomial are not distributed symmetrically about the origin. If all the roots have the same sign, then the coefficients $c_r$ follow the sign pattern

$$ \begin{align*} &+-+-+-+-\ldots \qquad \text{if all the roots are positive}, \\ &+\hspace{0.4pt}+\hspace{0.4pt}+\hspace{0.4pt}+\hspace{0.4pt}+\hspace{0.4pt}+\hspace{0.4pt}+\hspace{0.4pt}+\hspace{0.4pt} \ldots \qquad \text{if all the roots are negative}. \end{align*} $$

When m is a perfect square, most of the roots of $T^{\prime }_m(N,k)(x)$ are positive, and Corollary 3.2 shows that the coefficients of $T^{\prime }_m(N,k)(x)$ tend to this first pattern.

7 A conjecture on Hecke polynomial coefficients

In [Reference Rouse12, Conjecture 1.5], Rouse gave the generalized Lehmer conjecture: that for all $m\geq 1$ , N coprime to m, and $k = 12$ or $\geq 16$ , $\operatorname {\mathrm {Tr}} T_m(N,k) \neq 0$ . More recently, Clayton et al. [Reference Clayton, Dai, Ni, Xue and Zummo5, Conjecture 5.1] similarly conjectured that none of the Hecke polynomial coefficients vanish in the level one case. We propose the following conjecture that further extends both the generalized Lehmer conjecture, and [Reference Clayton, Dai, Ni, Xue and Zummo5, Conjecture 5.1]. The results in this article verify Conjecture 7.1 in all but finitely many cases.

We note that these lower bounds on k are the minimum possible. For any k less than these bounds, we will have $\dim S_k(\Gamma _0(1)) < r$ , and hence $c_r(m,1,k) = 0$ , trivially. Even relaxing the lower bound on k to just requiring that $\dim S_k(\Gamma _0(N)) \geq r$ will not work; Rouse [Reference Rouse12, Theorem 1.2] showed that for any given m and $k \in \{4,6,8,10,14\}$ , $\operatorname {\mathrm {Tr}} T_m(N,k) = 0$ for infinitely many N.

We now survey all the relevant previous results through the lens of Conjecture 7.1 (although they were not explicitly stated in these terms).

• • When $r\geq 1$ , $m=1$ , Conjecture 7.1 follows from the fact that $\dim S_k(\Gamma _0(N)) \geq r$ for $k=12r$ and $k\geq 12r+4$ .

• • In 2006, when $r=1$ and m is a non-square, Rouse [Reference Rouse12] showed Conjecture 7.1 for all but finitely many k, and for $100\%$ of N. When $r=1, m=2$ , he also completely verified Conjecture 7.1.

• • In 2022, when $r=1$ , $m=2$ , Chiriac and Jorza [Reference Chiriac and Jorza3] verified Conjecture 7.1 in the case of $N=1$ .

• • In 2023, when $r=2$ , $m\geq 2$ , Clayton et al. [Reference Clayton, Dai, Ni, Xue and Zummo5] showed Conjecture 7.1 in the case of $N=1$ for all but finitely many k. When $r=2$ , $m=2$ , they also completely verified Conjecture 7.1.

• • In 2023, when $r=1$ , $m=3$ , Chiriac et al. [Reference Chiriac, Kurzenhauser and Williams4] verified Conjecture 7.1 in the case of $N=1$ .

• • In 2024, when $r=2$ , $m\ge 2$ , we [Reference Ross and Xue11] showed Conjecture 7.1 for all but finitely many pairs $(N,k)$ . When $r=2$ , $m=3,4$ , we also completely verified Conjecture 7.1.

• • In 2024, when $r=2$ , $m\ge 2$ , Cason et al. [Reference Cason, Jim, Medlock, Ross, Vilardi and Xue1] showed a corresponding conjecture on the newspace $S_k^{\text {new}}(\Gamma _0(N))$ for all but finitely many pairs $(N,k)$ . When $r=2, m=2,4$ , they also completely verified the corresponding conjecture on the newspace.

• • In this article, when $r\geq 1$ and m is a square, Corollary 3.2 proves Conjecture 7.1 for all but finitely many pairs $(N,k)$ .

• • In this article, when r is even and m is a non-square, Corollary 4.2 proves Conjecture 7.1 for all but finitely many pairs $(N,k)$ .

• • In this article, when r is odd and m is a non-square, Corollary 4.3 shows that for k fixed, if Conjecture 7.1 holds for $r=1$ , then it also holds for each odd r for all but finitely many N. In particular, combining with Rouse’s result, this means that there exists a finite set K such that: (1) for all $k \not \in K$ , Conjecture 7.1 holds for all but finitely many N, and (2) even for $k \in K$ , Conjecture 7.1 holds for $100\%$ of N.

We observe that the last result listed here essentially reduces the problem of studying odd-indexed coefficients $c_r$ to just studying the trace, $-c_1$ .