1. Replications and cultural changes between then and now

Let us begin with another, related study. A little over 20 years ago, Downs and Shafir (Reference Downs and Shafir1999) investigated the role of compatibility and its impact on enriched versus impoverished options in the realm of social judgment. They presented participants with names of well-known personages with similar occupations, where, in each pair, respondents were more familiar with one personage than the other. For example, American participants were significantly more familiar with Ronald Reagan (an elder stateman at the time, who had been President of the United States a decade earlier) than with John Major (who had recently been the UK’s Prime Minister). Similarly, American participants were more familiar with Woody Allen (an American director at the height of his career at the time) than with Federico Fellini (a great Italian director barely known in the United States).

Participants were presented with various adjectives and had to select which personage in a pair was better described by each adjective. As predicted, and consistent with the compatibility hypothesis, participants were more likely to select the more familiar over the less familiar personage across opposite adjectives. For example, Reagan was judged as more confident than John Major by 61% of respondents, and as more insecure by 71%, totaling 132% across these opposing adjectives (with John Major receiving a total of 68%). Woody Allen received a total of 119% across confident and insecure, compared to Fellini’s 81%. José Canseco totaled 149% as compared to Tony Gwynn’s 51%, David Letterman totaled 139% compared to Kathy Lee Gifford’s 61%, and so forth. Not all judgments were quite this extreme, but the overall tendency of the more familiar personages, the ones offering more features compatible with the judgment, to be selected across opposite adjectives was pronounced and highly significant.

Now, how would a ‘very close replication’ of this study proceed? A ‘very close replication’ insists on using the same stimuli as in the original, with the claim that it is instructive to see how the phenomena persist through time. However, clearly the highly familiar items—Letterman, Canseco, and even Reagan—will not persist through time, or at least not to the same degree. The very hypothesis that generated these results in the late 20th century—namely, that participants will more easily find compatible instances in the familiar personages—predicts that those same stimuli will not replicate 30 or 40 or 50 years later, when those personages’ renown will have waned. (Furthermore, it is unlikely, e.g., that having married one adopted daughter and allegedly abused another, Woody Allen would retain his 1990s 70% ‘more moral’ advantage over Fellini.) The judgmental compatibility phenomenon published by Downs and Shafir in 1999 should, of course, replicate, but it would almost certainly require new stimuli: instead of Woody Allen and José Canseco, we may need Kate McKinnon and LeBron James.

Clearly, some phenomena, such as classical optical illusions, which are the outcome of evolutionary trends (Gregory, Reference Gregory2009) will replicate over long periods of time, whereas other phenomena, like those that depend on people’s attitudes toward Woody Allen or José Canseco, marriage, smoking, gender roles, or the environment, can change in just a few years. Where exactly to locate findings along this continuum is a theoretically interesting and nontrivial question. What is clear is that some findings are going to be time-sensitive in ways that optical illusions are not. And ‘very close replications’ need to observe those distinctions, or they risk generating confusing failures to replicate outdated items, rather than contributing to our understanding of the phenomena that lie behind them (for similar points, see, e.g., Ferguson et al., Reference Ferguson, Carter and Hassin2014; McGuire, Reference McGuire2013). This brings us to the issue at hand: replicating the patterns of choosing and rejecting documented by Shafir (Reference Shafir1993) three decades later.

2. Choosing and rejecting 30 years later

With the goal of conducting a ‘very close replication,’ Chandrashekar et al. (Reference Chandrashekar, Weber, Chan, Cho, Chu, Cheng and Feldman2021) chose to run the original materials used by Shafir (Reference Shafir1993). Both replication projects (Chandrashekar et al., Reference Chandrashekar, Weber, Chan, Cho, Chu, Cheng and Feldman2021; Klein et al., Reference Klein, Vianello, Hasselman, Adams, Adams, Alper and Batra2018) used Shafir’s Custody Problem. The original problem read as follows (with the original results, from Shafir, Reference Shafir1993, p. 549, reproduced below):

Original Problem:

Pilot testing when the Problem was first run had found that Parent A’s attributes were perceived as neutral, essentially offering no compelling reason to award or deny custody, whereas Parent B’s attributes were compatible with both choice and rejection. ‘Above-average income’ and ‘very close relationship with the child’ were highly positive (compatible with choice), whereas ‘lots of work-related travel’, and ‘minor health problems’ were much more negative (compatible with rejection; ‘extremely active social life’ in that context was close to neutral). In fact, Parent B’s rates of being awarded and denied (119%) exceeded the total of 100% expected if choosing and rejecting were complementary, z = 2.48, p < .02.

This problem, however, is not a tidy optical illusion—it is built on attributes that may very well change over 30 years. ‘Average working hours’ might sound more positive at a time when ‘more than ever, workers want to work fewer hours’ (Lufkin and Mudditt, Reference Lufkin and Mudditt2021), and ‘average income’ may sound more appealing in an era of rapidly increasing economic hardship that has left millions of Americans without adequate income to meet their basic needs. Similarly, as cultural norms change and perhaps grow more conservative, ‘extremely active social life’ may, three decades later, be judged more negatively, connoting a certain neglect of family life in favor of fun. ‘Lots of work-related travel’ was viewed, in the late 1980s, highly negatively. Subsequent decades, however, brought major changes, including the enormous increase in the frequency and popularity of work-related travel.Footnote ¹ When we collected ratings for various parental attributes in late 2017, we found that ‘lots of work-related travel’ was rated neutral—perceived as neither negative nor positive. But when we collected ratings again in the fall of 2022, perhaps due to the coronavirus pandemic and a shift to remote work, ‘lots of work-related travel’ was perceived negatively again.Footnote ² It is not always easy to predict how cultural change, unprecedented global crises like the pandemic, or, for that matter, some ‘fashionable’ associations, may change how people perceive certain stimuli three decades later. As a result, timely pilot testing will virtually always be a plus.

3. Pilot surveys

We conducted two preregistered pilot surveys to gauge attribute valence among online participants. Participants (Pilot Survey 1: n = 172 MTurkers, Pilot Survey 2: n = 164 MTurkers; recruited using CloudResearch; Litman et al., Reference Litman, Robinson and Abbercock2017) read the following instructions:

Participants were then presented with the original 10 attributes as well as several new attributes, and they rated each attribute’s valence on a scale from −5 (highly negative) to 0 (neutral) to 5 (highly positive). Both surveys followed this procedure; the second survey was used to rate additional attributes for new versions of the Custody Problem. We preregistered both Pilot Survey 1 (https://aspredicted.org/by4jy.pdf) and Pilot Survey 2 (https://aspredicted.org/my3ib.pdf) through AsPredicted.org. Data, materials, and analysis code for all studies are available on the Open Science Framework (https://osf.io/cxst6/). Ratings for all attributes from both surveys are presented in Table 1.

Table 1

Parental attribute valence ratings in Pilot Surveys 1 and 2

Note: Descriptive statistics—means (SDs)—from the two pilot surveys. Attributes were rated from −5 (highly negative) to 0 (neutral) to 5 (highly positive). The attribute ‘average working hours’ was rated twice in Pilot Study 1. The second time it was rated, the average rating was 1.98 (SD = 1.77).

As was to be expected, we found that the valence of some of the original attributes had changed. Interestingly, several attributes deemed neutral 30 years ago—average working hours, reasonable rapport with the child, relatively stable social life—were now quite positive. In fact, the impoverished parent, viewed neutrally three decades earlier, was viewed quite positively in 2022.

4. New (properly normed) versions of the Custody Problem

Based on the updated attribute ratings, we composed two new versions of the Custody Problem. These were close variations on the original problem with updated attributes intended to correct for changes in attribute perceptions over time. In the first new version, we ensured that the impoverished parent’s attributes (Parent A below, though actual order of presentation was counterbalanced) were all relatively neutral (absolute value of average valence ratings below .55). We further ensured that the enriched parent’s attributes (Parent B below) were either highly positively rated (three attributes rated 2.23 or higher) or negatively rated (two attributes rated −.92 or lower) in a combination roughly approximating the original enriched parent’s attribute ratings. The instructions were identical to those in Shafir (Reference Shafir1993) and reproduced in the context of the Original Problem above. For both versions, we followed Simonsohn’s (Reference Simonsohn2015) guidelines for powering replications by recruiting enough participants to ensure at least 2.5 times the original sample size (170), plus an additional cushion to detect even smaller effects.

We administered the first new Custody Problem in Study 1, preregistered through AsPredicted.org (https://aspredicted.org/n94mv.pdf) and run on Prolific. It is shown below, along with the percentage of participants who selected each option in the choose and the reject conditions (n = 552)Footnote ³ :

We found that the majority (71%) of participants asked to award custody awarded custody to the enriched parent, and the majority (57%) of participants asked to deny custody denied custody to the enriched parent. Mirroring Shafir’s (Reference Shafir1993) results, the sum of percentages of participants selecting the enriched option in the two conditions exceeds the 100% that would be expected if choosing and rejecting were complementary, instead totaling 128%, z = 6.64, p < .001. (Chandrashekar et al., Reference Chandrashekar, Weber, Chan, Cho, Chu, Cheng and Feldman2021, reported one-tailed tests; we follow their convention, but the p-values for this problem and the next problem are below .001 with both two-tailed and one-tailed tests.)

For the sake of robustness, we composed a second new version with a slightly different goal. Whereas the New Custody Problem 1 intentionally approximated the attribute valences in the original problem, it used a different set of attributes. In our second new version, we sought to more closely approximate the attributes used in the original, while still aiming to compose fairly balanced enriched and impoverished options. Accordingly, we assigned the impoverished parent (Parent A below) two of the original traits that were currently somewhat positively rated (‘average income’ and ‘average health’), while including one slightly negatively rated attribute (‘not very strict about household rules’) and keeping the other attributes close to neutral. We then composed the enriched option using the same attributes as in the original, with two attributes highly positively rated and one attribute highly negatively rated, along with two slightly negative attributes. We reasoned that revising the original impoverished parent’s currently highly positively rated attributes may allow us to retain the original enriched option and replicate a pattern similar to the original.

We administered the second new version in Study 2, preregistered through AsPredicted.org (https://aspredicted.org/fg7vn.pdf) and run on Prolific. The New Custody Problem 2 is shown below, along with the percentage of participants who selected each option in the choose and the reject conditions (n = 564)Footnote ⁴:

As before, and similar to the original, the sum of the percentages of participants who selected the enriched option in the two conditions exceeds the 100% expected if choosing and rejecting were complementary, instead totaling 119%, z = 4.40, p < .001. Both new versions of the Custody Problem yielded results that are consistent with the original findings of Shafir (Reference Shafir1993).

5. Concluding thoughts about ‘very close replications’

We began by gauging the valence of possible attributes to be assigned to the impoverished and enriched options in the Custody Problem, and we then composed two new versions of the problem that yielded results consistent with Shafir’s (Reference Shafir1993) findings. We found that the attributes used in the original Custody Problem had different valences three decades later, sufficient to render the original problem inadequate for testing Shafir’s original research question. In fact, the results of the ‘very close replications’ of Chandrashekar et al. (Reference Chandrashekar, Weber, Chan, Cho, Chu, Cheng and Feldman2021) and Klein et al. (Reference Klein, Vianello, Hasselman, Adams, Adams, Alper and Batra2018) show that the original problems run 30 years ago do not reproduce the original findings at present. (We also ran the original later in 2021 and failed to obtain the 1993 results, as reported in Cheek and Shafir Reference Cheek and Shafir2024.) Nonetheless, the original decision pattern documented in Shafir (Reference Shafir1993), with an asymmetry between choice and rejection occurring due to the enriched option obtaining more than its fair share relative to the impoverished option, appears alive and well.

We are not the first to raise concerns about the practice of simply readministering the same experimental materials in a sufficiently different context and then interpreting the different results as a sign of the initial findings’ unreliability (e.g., Crandall and Sherman, Reference Crandall and Sherman2016; Gergen, Reference Gergen1973; Schwarz and Strack, Reference Schwarz and Strack2014). Yet, the practice appears to persist without fully addressing these concerns. We view this as problematic and in need of refinement. When stimuli are context- or time- (or, obviously, language-) sensitive, running them ‘as is’, in different contexts, years later, in different cultural context, or in a language foreign to the respondents, without the appropriate updates or translations, is bound to mislead. From the Me Too movement to diets, commuting, bragging presidents, and family roles, enough has changed in the United States over the past 30 years to warrant a revision of the attributes used to describe relevant stimuli before certain replications ought to be explored. Otherwise, ‘very close replications’ do not seem, from a theoretical perspective, very close after all.

At a minimum, stimuli need to be revisited, in many circumstances preferably in collaboration with the original authors, to ensure that they afford a valid test of the original hypothesis (see also Fiedler et al., Reference Fiedler, McCaughey and Prager2021 on manipulation checks). In addition, researchers attempting replications need to discuss their findings with greater subtlety. It is problematic, in our view, to conclude that a decades-old finding ‘does not replicate’ without sufficient consideration for the different contexts (time, culture, norms, associations, etc.) in which the studies occurred. In some circumstances, all that replicators should be free to conclude is that the original materials, years later, or in different contexts, do not produce the same pattern of results, a finding that we find of limited insight by itself.

There is, furthermore, a concern that replications with little theoretical validity can impede research progress. Chandrashekar et al. (Reference Chandrashekar, Weber, Chan, Cho, Chu, Cheng and Feldman2021) used the original materials from Shafir (Reference Shafir1993) not only in an attempt at a replication, but also to address some theoretical work around the proposed mechanism of ‘accentuation’ (Wedell, Reference Wedell1997). Because of their faulty stimuli, however, it may be difficult to know what to make of the rest of their theoretical treatment.

In his analysis of choice–reject discrepancies, Shafir (Reference Shafir1993) attributed the observed patterns to the well-established principle of compatibility, where the weighting of inputs is enhanced by their compatibility with the response (see, e.g., Kornblum et al., Reference Kornblum, Hasbroucq and Osman1990; Proctor and Reeve, Reference Proctor and Reeve1990; Shafir, Reference Shafir1995; Slovic et al., Reference Slovic, Griffin, Tversky and Hogarth1990 for relevant discussions). Wedell (Reference Wedell1997) presented data in tension with this interpretation, and argued for an ‘accentuation’ hypothesis, according to which a greater demand for justification in choice compared with rejection leads to accentuation of the differences between alternatives in the choice condition. Cheek and Shafir (Reference Cheek and Shafir2024) document several more choose–reject patterns and discuss the contributions of both compatibility and accentuation, as well as the significant role of simple response errors, in contributing to the emergence of the choose–reject discrepancy.

We hope that this brief reply contributes both to basic research on decision-making and to more careful design and interpretation of replication studies. For now, what is clear is that the choose–reject patterns documented more than 30 years ago are, given the necessary updates, real and replicable.