No previous research has investigated the neural correlates of vocabulary acquisition in second language learners of sign language. The present study investigated whether poor vocabulary knowledge engaged similar prefrontal lexico-semantic regions as seen in unimodal L2 learners. Behavioral improvements in vocabulary knowledge in a cohort of M2L2 learners were quantified. Results indicated that there is significant increase in vocabulary knowledge after one semester, but stabilized in the second semester. A longitudinal fMRI analysis was implemented for a subset of learners who were followed for the entire 10 months during initial sign language acquisition. The results indicated that learners who had poor sign vocabulary knowledge consistently showed greater activation in regions involved in motor simulation, salience, biological motion and spatial processing, and lexico-semantic retrieval. In conclusion, poor vocabulary knowledge requires greater engagement of modality-independent and modality-dependent regions, which could account for behavioral evidence of difficulty in visual phonology processing.