The Stacking the Deck fallacy and its implications

The stacking the deck fallacy refers to the practice of focusing primarily or exclusively on the evidence that supports a particular premise, while disregarding, not engaging with, or not paying equal attention to contrary evidence. It is likely that the practice of stacking the deck contributes significantly to seemingly contradictory conclusions in bilingualism and neurocognition research. We submit that such a claim finds support in the findings of recent meta-analyses. Given that ample evidence suggests that bilingualism associates positive and negative outcomes (e.g., a finetuning of some aspects of the executive functions performance under at least some conditions alongside a general semantic fluency disadvantage), we suggest that it is meaningful to view these two types of effects through the lens of a trade-off approach: bilingualism, as an environmental-based trigger, confers both benefits and costs, because any enhancement is counterbalanced by a compensation, given an organism's finite resources. From this perspective, the presence of more and less positive associations with bilingualism is not a contradictory finding, but the counterbalancing effect of a trade-off.

Traced back to Darwin's observation that modifications in one part of an organism inevitably cause modifications in other parts, a recent integrative framework has provided a characterization of basic functional trade-offs in cognition (Del Giudice & Crespi, Reference Del Giudice and Crespi2018). Enhanced computation in one domain of the cognitive system is unlikely to occur without conferring a cost on another domain. Trade-offs are inherent to cognition and life in general; there is ample evidence of applications of the trade-off principle in other branches of neurocognitive research or evolutionary ecology (Garland, Downs & Ives, Reference Garland, Downs and Ives2022). Focusing on human cognition, semantic richness impacts creativity in the form of an originality-fluency trade-off (Beaty, Kenett, Hass & Schacter, Reference Beaty, Kenett, Hass and Schacter2022), speed of decision has been found to impact accuracy across a range of tasks (Del Giudice & Crespi, Reference Del Giudice and Crespi2018; Di Pisa, Kubota, Rothman & Marinis, Reference Di Pisa, Kubota, Rothman and Marinis2022), even cognitive and physical performance are organized in a trade-off relation (Longman, Stock & Wells, Reference Longman, Stock and Wells2017).

Set in these terms, the so-called advantageous bilingual associations are direct evidence of a cognitive enhancement and, crucially, indirect evidence for the cognitive costs such enhancement(s) entail. It is precisely this change of perspective that could, in principle, reconcile the results of many recent meta-analyses indicating indistinguishability from zero. If positive effects are systematically compensated by less positive or negative ones, the obtained balance will tend towards zero. Contrary to what some studies suggest (Paap et al., Reference Paap, Jonhson and Sawi2015), this is not evidence that the bilingual advantage hypothesis lacks the status of a robust or reliable phenomenon, but that it is incomplete: it simply does not describe the entire deck. Once we zoom out of individual effects that lean more and less towards a (dis)advantage interpretation and couch them within a trade-off approach, it becomes clear that each only provides half of the picture; the other half, which concerns the incurred compensations, sheds potential light on why some meta-analyses find a null effect, when averaging positive and negative effects.

This change of framing not only accounts for (i) the variation in the results and (ii) the meta-analytic null effect, but it can also shed light on the origins of the second mystery: After a century of research on bilingual cognition, why is an overarching theory that explains what drives these effects still missing? Even more important than the why question is the what question: what part of managing two (or more) languages confers these effects and under what conditions? Although several answers have been offered, all of them face challenges and have open issues/questions requiring dedicated research (cf. Blanco-Elorrieta & Caramazza, Reference Blanco-Elorrieta and Caramazza2021).

The road to progress entails identifying the main obstacles that one will likely face on the way. We argue that the most important obstacle is the lack of a clear picture as to what has prevented the field from developing ecologically valid theories. Our answer is that the stacking the deck fallacy impedes theory formation in the following way: those who find evidence for enhancements largely focus on positive evidence (i.e., results supporting the view that dual language management can result in mind/brain structure and functional adaptations). Similarly, those who find a disadvantageous or null effect often group these two categories together, mislabeling them as negative results, and accordingly focusing on them. In the end, this practice means that both “sides” put a spotlight on the findings that best support their position, inadvertently stacking the deck. Crucially, it is not our position that scholars with opposing positions ignore each other's results; they do engage with them but in different ways. This happens because the way one receives findings that are prima facie supporting or questioning their position in a debate is inevitably influenced by one's prior assumptions and viewpoints. In practice, this often leads opposing view-takers to interpret towards the same available evidence distinctly, directing one's focus to different sub-samples of the available evidence.

To provide concrete examples from opposing viewpoints, Paap and Greenberg (Reference Paap and Greenberg2013) test executive functions in bilinguals. In the critical Group x Condition interaction, of several anticipated patterns, they find only a single significant one that could be indicative of bilingualism-associated effects. The paper, however, is entitled ‘There is no coherent evidence for a bilingual advantage in executive processing’. In other words, a bunch of null effects and a single negative effect are presented as absence of (coherent) evidence for a bilingual advantage, instead of as sporadic evidence for a bilingual disadvantage. Of course, absence of evidence is not evidence of absence. And so, evidence of a cost in one domain in any given study does not constitute evidence against the potential for additive effects a priori. Similarly, studies that find evidence for a positive correlation between bilingualism and neurocognitive outcomes can and do draw particular attention to a single correlation that is not ubiquitously found in all administered tasks or in all tested bilingual populations. For instance, when discussing the clinical implications of their results, Desjardins, Barraza and Orozco (Reference Desjardins, Barraza and Orozco2019) emphasize the disadvantage in stream-segregation abilities, which was specific to the older bilingual group they tested. The young bilinguals’ advantage in the same domain is less analyzed, not entirely allowing for a full appreciation of the observed trade-off between enhanced L1 auditory processing in youth, and worsened L2 auditory processing in older adulthood.

To best contextualize the process of stacking the deck in the context of the available evidence in bilingualism research, let us consider, by analogy, the present challenge of two agents, A and B. A and B are given an opaque jar with the instruction to determine its contents. The jar contains blue and red objects that look like pencils, but A and B are oblivious to this fact. A and B take turns in taking objects out of the jar, examining them, and providing a hypothesis about the jar's contents. After having taken out some blue pencils, A reasonably suggests that the content of the jar is blue pencils. B has taken out of the jar some red pencils and counters that the content of the jar is red pencils. At some point, B does not manage to take anything out of the jar (i.e., null effect) and based on this they argue that since they failed to produce a blue pencil, A's blue pencil hypothesis lacks the status of a robust phenomenon. Then when it is again A's turn to take out an object, they surprisingly find a red one. This does not alter the fact that A has also found a good number of blue pencils before. In fact, their earlier premise “the jar contains blue pencils” is still true. However, it does not describe the full contents of the jar. Last, imagine that unexpectedly B takes out of the jar a blue pencil. After observing it, they argue that this is not proof that A's blue pencil hypothesis is correct, because the effect size of blueness in this specific pencil is not sufficient; this pencil looks purple to B and purple contains red too, so this pencil may be taken as showing that the red pencil hypothesis is on the right track. Eventually, A and B describe the contents of the jar in two different ways, based on seemingly contradictory results, such that they are not able to form an overarching Theory of the Jar. This happens because A and B inadvertently stack the deck: They focus on and explain specific pencil-selection events, without tackling the entire range of them in the same way. While A focuses on finding evidence for the blue pencil hypothesis, B does the same, but guided by the implications of the red pencil approach. Crucially, these differences in their priors are not insignificant. The circle of scientific experimentation always starts from a set of theoretically informed hypotheses that crystallize into research questions that are then put to test. Fragments of some theory are always present at the initial stages of a scientific investigation and claims of fully atheoretical testing or theory-free data gathering should be viewed with some scepticism (Ludlow, Reference Ludlow, Kertész, Moravcsik and Rákosi2019).

Where does this picture leave us in terms of bilingual effects? We make two important claims. First, we argue that both A and B are largely right in terms of the reliability of the adduced evidence that supports their claims. In other words, we suggest that the outcome of all pencil-selection events is true at the same time, regardless of whether they have been used to support the blue or the red pencil hypothesis. This is not a trivial point, because it goes against the mainstream view that the field consists of contradictory results. Under this scenario, what gives the impression of contradictory or inconsistent results is the fact that different scholars arrive at different versions of the same ‘truth’ because they engage with the results in a theoretically informed way that contains traces of red and blue, and subsequently colors their approach to the results accordingly. The different color, however, is a matter of different interpretation; it is not a contradiction that is inherent in the evidence. We are thus led to the conclusion that different versions of the same truth hold and are held at the same time.

Second, we suggest that both A and B have stacked the deck not only due to their different priors and guiding hypotheses, but also because they have employed an “either blue or red” approach. To understand this second point, let us imagine that the owner of the jar appears and explains that all the objects taken out of the jar are both blue and red, but (a) the way they were handled caused them to break, giving the false impression of separate blue and red pencils; and (b) the different theoretical light under which A and B were observing them has affected their perception – such that A sees blue pencils (positive effects) and B sees red pencils (negative effects). Under a trade-off approach, what A and B thought of as separate pencils may in reality be chunks of a single block of wood with a continuum of colors that ranges from blue to red. Importantly, we do not propose a mere shift of perspective that boils down to going from “either blue or red” to “blue and red”; we crucially establish a shared origin, making the claim “by nature both blue and red, because blue/red continuum”. Put another way, enhancements and costs may be observed in isolation, and indeed they may give the impression of separate effects. Yet once we zoom out from specific tests and measures, and observe the overall adaptations to bilingualism across cognitive domains at the population level, it becomes clear that positive outcomes and negative outcomes complement each other as parts of a trade-off of effects. The upshot is that bilingualism may trigger cognitive enhancements, but these are necessarily linked to some compensatory ramifications, due to functional conflicts between the allocation of the organism's finite resources across different domains.

The spectrum of bilingual trade-offs

What is missing for developing an overarching theory of neurocognitive adaptations to bilingual experience is a unifying perspective that reconciles the seemingly contradictory results. We suggest that the necessary shift in perspective the field needs entails the coupling together of two critical notions: spectrum and trade-off. Bilingualism, as a spectrum of experience (e.g., DeLuca et al., Reference DeLuca, Rothman, Bialystok and Pliatsikas2019; Luk & Bialystok, Reference Luk and Bialystok2013), confers certain neurocognitive adaptations that in turn produce outcomes that form a continuum of trade-off relationships. In this sense, potential cognitive enhancements necessarily entail compensations, questioning any value to debates or discussions whose framing is predicated on an expectation that replication would not be contingent on the context of specific bilingualism factors or be universal in how effects present in all potential domains. The trade-off hypothesis can be conceived of as consisting of different ramifications that respond to an environmental trigger (in this case, bilingualism) through playing a zero-sum game. Once we zoom out of individual studies and observe the overall picture, positive and negative effects seem to go hand in hand, forming a balance that regresses to the baseline. This does not mean that the effects behind what the field often refers to as ‘bilingual advantages’ and ‘bilingual disadvantages’ are incorrect or unreliable. It simply follows that such labels are misnomers not simply because they are too reductionist, but rather they are further uninformative because they do not recognize that whatever positive or negative value can be ascribed to them in isolation only acknowledges half of the picture.

To better understand the ecological validity of a bilingual trade-off approach, let us examine an example where we can zoom out from specific studies and proceed to observing effects at the population level. Let us compare two sets of studies that target different bilingual populations: Baus, Santesteban, Runnqvist, Strijkers and Costa (Reference Baus, Santesteban, Runnqvist, Strijkers and Costa2020) and Antón, Carreiras and Duñabeitia (Reference Antón, Carreiras and Duñabeitia2019) test young adult bilingual speakers of Basque–Spanish (set I), and Sadat, Martin, Alario and Costa (Reference Sadat, Martin, Alario and Costa2012) and Hernández, Costa and Humphreys (Reference Hernández, Costa and Humphreys2012) test young adult bilingual speakers of Catalan–Spanish (set II). Set I found a bilingual disadvantage in picture naming (Baus et al., Reference Baus, Santesteban, Runnqvist, Strijkers and Costa2020) and a bilingual advantage in a subset of tasks that tap into working memory (Antón et al., Reference Antón, Carreiras and Duñabeitia2019). The tentative conclusion from set I is that a bilingual disadvantage in lexical production may be compensated by enhanced working memory. Set II found a bilingual disadvantage in picture naming (Sadat et al., Reference Sadat, Martin, Alario and Costa2012) and a bilingual advantage in disengaging attentional resources from irrelevant information held in working memory (Hernández et al., Reference Hernández, Costa and Humphreys2012). Again, we observe that while lexical access may be hampered in some bilinguals, this disadvantage is offset by an enhancement in the deployment of working memory-mediated resources. Although if one zooms into any of these four studies in isolation, one will likely get the impression of isolated positive and negative effects, when viewing them together, it becomes clear that the two tested populations show strong similarities in terms of the outcome: effects that enhance one part of the system are counterbalanced by a cost in another part. What looks like either red or blue pencils (giving the false impression of contradictory findings) may be alternatively conceived of as different chunks of a continuum of cascading effects that range from blue to red.

Overall, this shift of perspective from individual effects to trade-offs that form a continuum of outcomes offers three critical take-home points. The first relates to the reliability of the available evidence that supports positive and negative outcomes in isolation. In line with Grundy (Reference Grundy2020), we suggest that the large volume of results acquired by the proponents of different theories within bilingualism research is reliable and largely free of Type I errors, such that it is meaningful to ask when and under what conditions effects occur, and not whether or not they ever do.

The second point has to do with the empirical evidence that justifies the shift towards the trade-off approach. Although most studies do not explicitly acknowledge that negative results do not occur independently of positive results, there is evidence to back up this shift of perspective. Several studies describe compensatory effects within populations, leaving open the possibility that whatever mechanism is responsible for enhancements could also be responsible for the costs (Krizman, Bradlow, Lam & Kraus, Reference Krizman, Bradlow, Lam and Kraus2016; Leivada, Mitrofanova & Westergaard, Reference Leivada, Mitrofanova and Westergaard2021a; Tao, Taft & Gollan, Reference Tao, Taft and Gollan2015).

Third, this shift of perspective carries implications for meta-analyses and their employed classification systems, but also for how the field conceptualizes positive and negative outcomes. Under the trade-off approach, a null result is not synonymous with a negative result. Also, a negative result is not evidence against the possibility of positive results in another domain. If enhancements are compensated by costs, the practice of grouping negative and null results in one ‘evidence against the bilingual advantage’ mega-category (e.g., de Bruin, Treccani & Della Sala, Reference de Bruin, Treccani and Della Sala2015) is less reasonable than grouping positive and negative outcomes vs. null results, because positive and negative outcomes entail each other. Put another way, treating a negative result as synonymous to a null result, and putting them together in one category does not do justice to the correlation between enhancements and costs. Similarly, publication biases that favor the publication of positive results (de Bruin et al., Reference de Bruin, Treccani and Della Sala2015) operate on a similar view of positive and negative outcomes as separate constructs. This misconception contributes towards stacking the deck, because it permits granting a prominent position to a part of the available evidence (i.e., either the blue or the red pencils in the earlier jar metaphor), ignoring the shared origin of the two.

Although many studies have demonstrated bilingualism-related costs and benefits within language and neurocognition, very few and far between are papers that discuss the linguistic and neurocognitive dimensions together. There is a decoupling of sorts between linguistic and neurocognitive research, and while both dimensions converge in their views that experience matters deterministically for linguistic outcomes and neurocognitive adaptations, very rarely do arguments that speak directly to them suggest that outcomes within these two dimensions might be directly related in one loop of trade-offs. This argumentation then suggests a program in which neurocognitive studies should include more fine-grained linguistic measures; not mere proxies for overall proficiency, but rather specific domains of grammar at a higher level (morphosyntax, discourse pragmatics, information structure) that can be a priori determined as part of the trade-offs one might expect if a hypothesized domain of cognition is affected. The same is true in reverse: processing studies should include more measures of neurocognitive functions, not simply to probe for individual differences per se (e.g., if one has enhanced working memory, X domain of grammar would be affected), but in strict coherence with how exponents of bilingualism engagement would be hypothesized to affect working memory, which in turn would have a proportional effect for particular domains of grammar in (measurable) accord.

In conclusion, failing to appreciate how both positive and negative effects co-occur in a population as part of a trade-off means that we are doomed to focus on only half of the picture, while probably thinking that we do justice to the full range of results. Indeed, a trade-off approach considers it necessary for research to unpack the dynamic relationship between dual language experience and individual differences in associated mind/brain adaptations. And yet, it views the trend of treating bilingualism as a spectrum of differentially deterministic experiences in recent work as merely one of several needed steps in the right direction. Because it anticipates a yin for every yang, the next step will be to determine specific pairings of domains with predictable trade-off relationships. For example, is it really the case that decreased lexical access in many bilinguals is consistently offset by enhancements in working memory-mediated resources? If so, why? What is it about these two domains that makes them good candidates to pair-together in a trade-off loop? To be sure, work is needed – theoretical and empirical – to make reasonable and testable claims about (the entire set of) specific neurocognitive and linguistic domains that have trade-off relationships. Suffice it to say, for now, there are significant benefits in conceptualizing the neurocognitive study of bilingualism in terms of trade-offs. Most promisingly, doing so might very well make it possible to accommodate without any concessions or compromising from anyone what have been argued to be incompatible and irreconcilable claims/conclusions made on the basis of the same available data.