Recent research suggests that a cognitive bias, the illusion of causality, can be attenuated when the task is presented in a foreign language (Díaz-Lago & Matute, 2019a, Quarterly Journal of Experimental Psychology, 72(1), 41–51), supporting the well-known foreign language effect on decision making and reasoning. We conducted a replication study with a large sample (N = 220), determined through a Bayes factor design analysis, but our results did not support the original findings. This finding challenges the generalizability of the foreign language effect on reducing cognitive biases. Additionally, we found that the magnitude of the illusion decreased with increasing years of formal education and was generally weaker among male participants compared to females. These findings emphasize the importance of using samples with balanced demographic characteristics to avoid potential confounds in between-group comparisons. Overall, our study highlights the need for further research to clarify the conditions under which the foreign language effect can influence cognitive biases.