This study investigates whether individual differences (IDs) in speakers’ perceptions of psycholinguistic dimensions (familiarity, knowledge, transparency, ambiguity, valence, arousal) predict idiom processing advantage over matched novel control phrases in L1 and L2 speakers in a phrase judgment task. Bayesian multilevel models showed that item-level average norms from L1 speakers for familiarity, valence, and arousal facilitated idiom judgment in both groups, whereas transparency and ambiguity had inhibitory effects. Beyond item means, deviations in speaker-specific perceptions from the item-level average norm predicted idiom judgment advantage. In both groups, individuals who rated an idiom as more positive and more ambiguous than its item-level average norm showed larger idiom judgment advantages. In L1–L2 comparisons, individual-level deviation effects of valence and transparency were stronger in L2 than in L1 speakers, whereas ambiguity deviation effects were stronger in L1 speakers. Overall, modeling ID effects beyond group norms reveals finer-grained L1–L2 differences in language processing.