Speech disfluency in first language (L1) and second language (L2) is not only quantitatively different (e.g., number and length of pauses) but also qualitatively different (e.g., distribution of pauses); however, how pause distributions in L1 and L2 affect judgments of perceived fluency is unclear. The study examined the relationship between pause phenomena and perceived fluency using two experiments. Experiment 1 investigated the relative contributions of frequency, length, and distribution of silent pauses to perceived fluency of L2 speech. Experiment 2 tested causal effects of pause location on perceived fluency of L1 and L2 speech using phonetic manipulations. Findings suggest a significant role of pause location in perceived fluency. In Experiment 1, silent pause rate within a clause demonstrated the strongest correlation with L2 fluency ratings, and in Experiment 2, perceived fluency of L1 and L2 speech was influenced by pause location. The findings suggest that listeners seem to be sensitive to pause location and to understand that pauses within clauses tend to reflect reduced cognitive fluency.