Listeners flexibly adapt utterance interpretation based on speaker characteristics, but less is known about how this works in multiparty conversations, particularly for pragmatic inferences like Mutual Exclusivity (MEI) – the tendency to map novel terms onto unnamed referents. MEI may reflect a flexible, speaker-sensitive process or a rigid default. This study tested whether listeners modulate MEI based on speaker-specific referential consistency in multiparty conversations. Two eye-tracking experiments were conducted (Exp 1: N = 32, English; Exp 2: N = 44, Spanish), where participants followed instructions from two speakers. In experimental conditions, one speaker was consistent (reused labels), while the other was inconsistent (sometimes relabeled objects). Control conditions featured two consistent speakers. Results showed robust and early MEI across conditions. No speaker-specific adaptation occurred, but Experiment 1 showed context-general adaptation: exposure to an inconsistent speaker within a conversation reduced MEI overall. Findings suggest that in high-demand communicative contexts, MEI operates as a strong default heuristic, largely resistant to speaker-level inconsistency. Evidence for sensitivity to broader contextual reliability was observed in Experiment 1 but did not replicate in Experiment 2, leaving open whether MEI exhibits limited contextual flexibility or whether the initial finding reflected design-specific factors.