Three years ago, in the January 2015 issue of Perspectives on Psychological Science, Davidai and Gilovich argued that Americans tend to overestimate socio-economic prospects for the poor. Their evidence: when asked to estimate the likelihood that a child born into a low-income family in the United States would secure a higher social class ranking in adulthood, participants in Davidai and Gilovich’s study (N = 3,324 online survey respondents) exaggerated the actual likelihood of ascent by an average of ten percentage points. Four weeks later, in the February 2015 issue of Psychological Science, we (Reference BloomeChambers, Swan & Heesacker, 2015) argued precisely the opposite — our research participants (N = 865) had apparently underestimated the very same upward-mobility statistic.

Seven months ago, here in Judgment and Decision Making, we (Reference Swan, Chambers, Heesacker and NeroSwan, Chambers, Heesacker & Nero, 2017) recanted — after (a) cataloging the slew of ostensibly superficial differences between the two teams’ methodologies and (b) presenting evidence that just one of those differences could account for the discrepancy between the two teams’ findings, we concluded that the probability of a false positive result produced by systematic measurement error, such as question-wording artifacts, was high enough to undermine both teams’ claims about population-level perceptual (in)accuracy.Footnote ¹

Now, Davidai and Gilovich (2018; this issue) suggest that the error was endemic exclusively to our questionnaire wording. In brief: Reference Davidai and GilovichDavidai and Gilovich (2015) had asked their participants to estimate socio-economic mobility rates by dividing the population into quintiles, whereas we (Reference BloomeChambers et al., 2015) had asked had our participants to divide into tertiles. When we inverted the quintile-tertile factor in a new set of experiments (Swan et al., 2017; N = 1,142) while holding all other differences in wording constant, the overestimation and underestimation effects observed respectively by Reference Davidai and GilovichDavidai and Gilovich (2015) and Chambers et al. (2015) either vanished or reversed (see Table 3 in Swan et al., 2017). Our quintile-tertile-inversion results could not bear on the question of which format is more likely to capture respondents’ true perceptions, but we did speculate that the tertile format may better reflect how people naturally think about income distributions (in terms of the upper, middle, and lower classes).

We may stand corrected. In three new studies (N = 392), Reference Davidai and GilovichDavidai and Gilovich (2018) found that most people sampled divided the income distribution pie into five (or more, not three) slices. Consequently, on Davidai and Gilovich’s account, when we (Reference BloomeChambers et al., 2015) asked our study participants to estimate upward mobility from the bottom third — a non-intuitive category label — we exposed ourselves to a set of tertile-specific measurement vulnerabilities. Namely, in two additional studies (N = 304), Reference Davidai and GilovichDavidai and Gilovich (2018) found that people tend to (a) assume that a three-rung ladder is ipso facto harder to climb than a five-rung ladder; and (b) estimate less upward social mobility when a questionnaire first asks about (primes the concept of) immobility (as ours did).Footnote ² Citing additional evidence that people normally conceptualize movement in rankings as the likelihood of rising up rather than as the likelihood of staying put (Reference Chetty, Hendren, Kline, Saez and TurnerDavidai & Gilovich, 2016), Reference Davidai and GilovichDavidai and Gilovich (2018) concluded that their original (2015) quintile prompts elicited reliable and valid responses (whereas our original tertile prompts did not).

Although we find Davidai and Gilovich’s (2018) new data to be interesting, we do not believe that they support their central claim: that the average American overestimates the actual number of people who manage to transcend childhood poverty in the United States. We dedicated most of the text in our (2017) critique to an experimental demonstration of the quintile-tertile effect, but, as we noted in our (2017) introduction, the key distinction between the Reference Davidai and GilovichDavidai and Gilovich (2015) and Chambers et al. (2015) investigations was simpler: the two teams had applied different true-mobility rate comparators. Whereas Reference Davidai and GilovichDavidai and Gilovich (2015) had weighed their participants’ guesses against data from the Panel Study of Income Dynamics (PSID) as reported by the Pew Economic Mobility Project (2012), we (Reference BloomeChambers et al., 2015) had relied on tax-record data curated by Reference Chetty, Hendren, Kline, Saez and TurnerChetty, Hendren, Kline, Saez and Turner (2014).Footnote ³ The distance between the two comparators matters: when one contrasts Chetty et al.’s true-mobility-rate-point-estimate (rather than Pew’s) with Davidai and Gilovich’s (2015) participant-guess-averages, the overestimation-of-mobility effect disappears entirely. To wit, if Davidai and Gilovich had used Chetty et al.’s (2014) data to compute true-mobility-rate benchmarks in their original (2015) study, they would have discovered that their participants perceived social mobility trends with dead-on accuracy.Footnote ⁴ Thus, even if one accepts that quintile measures yield psychometrically superior data, Davidai and Glivoch’s (2018) overestimation claim still depends on the belief that Pew’s (2012) point estimates are more accurate than Chetty et al.’s (2014).

Our (2017) discovery of elicitation-method sensitivity (the quintile-tertile effect) alone would not have led us to conclude that the quest to quantify population-level perceptual (in)accuracy is hopeless. But reckoning with both of these problems — (a) comparators with unknown error margins weighed against participants’ responses to (b) estimation prompts with a high potential for subtle systematic bias — did move us to emphasize the formidable methodological challenges that face researchers who approach this topic. The first problem (selecting the correct comparator) does not appear to have a single econometric solution (e.g., see Bloome, 2015; and Reference WarrenWarren, 2014). Comparators should therefore be accompanied by clear justifications and statements of limitations.

Reference Davidai and GilovichDavidai and Gilovich (2018) seem to believe that they have solved the second problem, that of ensuring that mobility-perception prompts avoid hidden question-wording confounds. We respectfully disagree — even if the evidence continues to favor the use of quintile response formats, other possible complications remain. For instance, Davidai and Gilovich’s (2015) questionnaire apparently asked participants to forecast mobility rates moving forward — the likelihood that a person born today would end up in a given income group in the future. Both Pew (2012) and Chetty et al. (2014) approximated upward social mobility trends in the past. Thus, unless one assumes that intergenerational elasticity in earnings will remain stable over the next several decades (this seems highly unlikely to us; see Reference Chetty, Hendren, Kline, Saez and TurnerChetty et al., 2014; and Reference Mishel, Bivens, Gould and ShierholzMishel, Bivens, Gould & Shierholz, 2012), arguably there is no valid comparator for Davidai and Gilovich’s (2015) data. It may be that most participants in Davidai and Gilovich’s (2015) study ignored these temporal cues and estimated mobility rates in the past, but without a direct empirical test, in our view it remains prudent to doubt Davidai and Gilovich’s (2018) broad conclusion that “Americans indeed overestimate the [current] degree of mobility in the United States.”

Of course, there are other ways to assess trends in population-level perceptual (in)accuracy. Consider, for instance, Reference Kraus and TanKraus and Tan (2015) — two weeks after Reference Davidai and GilovichDavidai and Gilovich (2015) and two weeks before us (Reference BloomeChambers et al., 2015), Kraus and Tan also published a paper (in the Journal of Experimental Social Psychology) on the topic of how Americans perceive the degree of socio-economic mobility in the United States. Like Reference Davidai and GilovichDavidai and Gilovich (2015), Kraus and Tan asked their online survey respondents (Study 1 N = 751) to guess the future social class rankings of people born into the bottom income quintile, and, like Davidai and Gilovich, Kraus and Tan observed undue optimism. (See also Kraus, 2015.) But Reference Kraus and TanKraus and Tan (2015) also asked their study participants to estimate several more specific social mobility statistics. For instance: how many people out of 100 during a ten-year time period from 1997–2006 would move up from the bottom 20% of income (a) by working 1000 extra hours or (b) with some kind of college degree? People overestimated on both counts.

Should we treat Kraus and Tan’s (2015) findings as a line of converging independent evidence that, despite the methodological concerns we have raised about Davidai and Gilovich’s (2015) and Chambers et al.’s (2015) studies, Americans really do overestimate social class mobility? Again, the comparator problem rears its head — Kraus and Tan drew their benchmark values from yet another data source (Reference Mishel, Bivens, Gould and ShierholzMishel, Bivens, Gould & Shierholz, 2012). Kraus and Tan’s real-mobility-rate estimates were more pessimistic: citing data from the PSID, they claim that fewer than 1% of people born into the bottom 20% will rise to the top 20% in adulthood. (Chetty et al. estimated the number closer to 8%.) The consistency of Kraus and Tan’s participants’ responses across a diverse array of questions may well signify an underlying socio-cognitive phenomenon,Footnote ⁵ but, in our view, without a clear justification for the use of these data as the comparator, the title of Kraus and Tan’s (2015) paper, Americans Overestimate Social Class Mobility (see also Kraus’ 2015 pre-registered replication report, Americans Still Overestimate Social Class Mobility) remains debatable.Footnote ⁶

Of course, there are still other ways to assess trends in population-level perceptual (in)accuracy. In a final pair of studies (N = 206), Reference Davidai and GilovichDavidai and Gilovich (2018) observed that Americans tend to rank the United States too favorably among a group of fifteen countries with varied social class mobility rates. Does the preponderance of evidence now support the broad claim that Americans’ overestimate social class mobility? Alas, although we may update our priors, our skepticism endures. In this case, we note that people’s estimates of social mobility rates within a country may be orthogonal to their estimates of social mobility rankings between countries. One can imagine, for instance, that while most Americans believe their country boasts more socio-economic mobility than most others (which is demonstrably not the case), opportunities for advancement are still unfairly constrained (which is demonstrably the case). Moreover, in our 2017 critique we developed our own novel measure of beliefs about economic mobility — when we asked our participants (N = 722) to pick from between three graphical representations of (pictures of ladders overlaid with) different social-mobility-rate distributions, the results could not have been clearer: the vast majority of our participants (72%) underestimated even when they were (a) presented with quintiles and (b) could have selected a distribution that corresponds exactly to the average participant guesses (overestimates) from Davidai and Gilovich’s (2015) original study. Reference Davidai and GilovichDavidai and Gilovich (2018) did not mention or discuss this finding. Yet another team of researchers (Reference Alesina, Stantcheva and TesoAlesina, Stantcheva & Teso, 2018) asked a slightly different question, used Chetty et al.,’s (2014) data as their comparator, and documented a trend of accuracy.

When we advised our judgment-and-decision making colleagues to shift away from the question of population-level (in)accuracy, we were admitting our own discouragement in the face of so many granular complexities. Davidai and Gilovich respectfully disagreed, and we are glad they did. Indeed, effective policy development depends on our ability to discern these trends. And so, in the end, we concur — the quest to measure distortions in Americans’ socio-economic-mobility perceptions deserves scientific attention.Footnote ⁷ Our step-one proposal is to select and justify a fair comparator (with appropriate error bars).