We determine all the extremal Gromov-Witten invariants of the Hilbert scheme of $3$ points on a smooth projective complex surface. Our result for the genus- $1$ case verifies a conjecture that we propose for the genus- $1$ extremal Gromov-Witten invariant of the Hilbert scheme of n points with n being arbitrary. The main ideas in the proofs are to use geometric arguments involving the cosection localization theory of Kiem and J. Li [17, 23], algebraic manipulations related to the Heisenberg operators of Grojnowski [13] and Nakajima [34], and the virtual localization formulas of Gromov-Witten theory [12, 20, 30].