Hurwitz numbers, which count certain covers of the projective line (or, equivalently, factorizations of permutations into transpositions), have been extensively studied for over a century. The Gromov-Witten potential F of a point, the generating series for descendent integrals on the moduli space of curves, is a central object of study in Gromov-Witten theory. We define a slightly enriched Gromov-Witten potential G (including integrals involving one ‘$\lambda$-class’), and show that, after a non-trivial change of variables, G = H in positive genus, where H is a generating series for Hurwitz numbers. We prove a conjecture of Goulden and Jackson on higher genus Hurwitz numbers, which turns out to be an analogue of a genus expansion ansatz of Itzykson and Zuber. As consequences, we have new combinatorial constraints on F, and a much more direct proof of the ansatz of Itzykson and Zuber. We can produce recursions and explicit formulas for Hurwitz numbers; the algorithm presented proves all such recursions. As examples we present surprisingly simple new recursions in genus 0 to 3. Similar recursions should exist for all genera. As we expect this paper also to be of interest to combinatorialists, we have tried to make it as self-contained as possible, including reviewing some results and definitions well known in algebraic and symplectic geometry, and mathematical physics. 2000 Mathematical Subject Classification: primary 14H10, 81T40; secondary 05C30, 58D29.