2.1. Reporting practices for quantitative results in JDM

Based on the proportions shown in the upper panel of Table 3, and their relatively narrow confidence intervals, we can draw some reasonably precise conclusions about reporting practices in recent JDM research articles. First, it may surprise readers that nearly 10% of the research articles that we sampled—all of which reported quantitative research—did not report a single p-value. Certainly, this was a surprise to us because we had expected this to be very rare indeed. We scrutinized the contents of these 14 ‘no p-value’ articles to understand how they analyzed and reached conclusions about their quantitative data (summarized in Table S5 in the Supplementary Material). Eleven articles used computational modeling of one kind or another, and several reported confidence/credibility intervals or error bars. Further exploration found that, compared to articles reporting p-values, the no p-value articles were somewhat less likely to report standardized effect sizes and power analyses, or be published in an economic focus journal, and were more likely to use Bayesian methods (Table S7 in the Supplementary Material). There was at least one no p-value article from each of the 6 journals we examined, with 6/14 published in Decision. Thus, clearly, statistical inference using p-values is not a strict pre-requisite for JDM publications. Nonetheless, NHST survives (and perhaps thrives) in JDM research. Yet, for the most part, NHST is employed within a mixed economy alongside other quantitative tools and approaches, as illustrated by the fact that only 17 of the 150 articles we reviewed reported a p-value but no standardized effect size, confidence interval or SE bars (0.11, 95% [0.07, 0.17]). So, while there may be no full-scale rejection of NHST as some have advocated (for some advocates, see Fidler, Reference Fidler2002), our data may offer 2 crumbs of comfort to those who favor approaches to inference that do not rely on null hypothesis testing. First, quantitative research using (exclusively) non-NHST approaches does get published in JDM journals. And second, there is little evidence of sole reliance on reporting p-values without any estimates of effects being reported.

Next, the reporting of effect size information seems to be reasonably common. From our data, we estimate that around 7 in 10 recent JDM research articles report at least one standardized measure of effect. Given that some of the complementary set of 3-in-10 articles report unstandardized effect size information in ways that are meaningful to readers,Footnote ⁵ we think this means that the field is aligning reasonably well with the APA Task Force recommendation to ‘always present effect sizes for primary outcomes’ (Wilkinson et al., Reference Wilkinson1999, p. 599).

Third, however, the field seems not so well aligned with that same Task Force’s recommendation that ‘interval estimates should be given for any effect sizes involving principal outcomes’ (p. 599) or Cohen’s (Reference Cohen1994) recommendation to ‘routinely report effect sizes in the form of confidence limits’ (p. 1002). Our analysis suggests that only 4 or 5 in every 10 recent JDM research articles report at least one confidence interval. If one extends the ‘interval estimate’ definition to include graphical SE bars, the proportion of papers reporting at least one interval estimate rises a little. In our data, there were 27 of the 150 articles that presented figures with SE bars without reporting any true confidence interval, meaning that 0.63 of the sample (95/150) reported at least one CI or figure with SE bars (95% CI [0.55, 0.71]). However, for these data, it is worth noting that intervals defined by estimate ± SE lack the precision of a confidence interval because the percent confidence associated with the interval is not known to the reader unless the sample is very large. Moreover, error bar presentations are often not displaying an interval for the effect of primary interest (Cumming, Reference Cumming2014) as when means ± 1 SE are shown when mean differences are the phenomenon of interest.

Aggregating the data for ES and CI in Table 3 together, it seems that—to a reasonable degree—reporting practices in JDM research reflect an estimation perspective (Cohen, Reference Cohen1994; Cumming, Reference Cumming2014). Only 20 of the 150 articles we reviewed did not report any standardized effect size, any confidence interval or use SE bars (0.13, 95% CI [0.09, 0.19]).

By themselves, the fine-grained data on which ‘families’ or types of effect size measure or confidence interval are reported hold limited interest (Table 3). For example, the fact that 99 articles reported an r-family effect size while only 32 reported a d-family effect size could simply reflect that correlation and regression techniques are wonderfully flexible and widely used. However, a comparison of the columns for ES and CI in Table 3 reveals a curious quirk. Among articles that report a standardized effect size, substantially more report an r-family effect than report a d-family effect (91% vs. 29%). This between-family difference reverses for confidence interval reporting, wherein CIs for r-family effects are less common than CIs for d-family effects (37% vs. 66%). We can only speculate as to the reasons for these discrepant patterns—a different kind of study to ours would be needed to uncover whether, for example, researchers focus on different kinds of statistical concept for different families of effect; whether different traditions of research design or data analysis have distinct reporting conventions; or whether something as mundane as the defaults within commonly used software packages make it more or less likely that a standardized ES or a CI is generated for particular kinds of analyses.

2.2. The discussion of findings in JDM research

For 60 years or more, researchers have been documenting the reporting practices of research articles in the behavioral sciences (e.g., Sterling, Reference Sterling1959). Therefore, our review represents just one more offering in this tradition, albeit one that, we hope, provides useful information about the reporting of results in a specific field (JDM) at a specific point in time (publication year 2023). What may be somewhat unusual is our systematic analysis of how results are discussed in the narrative sections of research articles—though see Finch et al. (Reference Finch, Cumming, Williams, Palmer, Griffith, Alders, Anderson and Goodman2004) who, like us, examined whether authors discussed confidence intervals and error bars. The most noteworthy findings of our analysis can be seen in the middle panel of Table 3, and in the discrepancies revealed by comparing the upper and middle panels of that table.

Most striking is the near-complete absence of discussion of the information about the uncertainty or precision of estimation, or estimation range, that is contained within a confidence or credibility interval. Thus, while we identified 68 articles that reported at least one confidence interval, we identified only 4 articles that made explicit reference to some feature of a confidence interval or SE bar in the Discussion/Conclusion section of the article. Because this practice was so rare, it is feasible to present all 4 text extracts so coded in Table 4. Our data for discussing CIs point to the same conclusion given by Finch et al. (Reference Finch, Cumming, Williams, Palmer, Griffith, Alders, Anderson and Goodman2004) who analyzed 696 articles published 1990–2000 in Memory & Cognition (of which 30% reported confidence intervals or error bars on figures):

Table 4

Extracts from all 4 articles coded as having discussed confidence intervals in words or numbers

Note: Row number refers to the master spreadsheet of our data available at osf.io/apfg8/.

Of course, because our coding exercise was both a vigilance task (requiring careful reading) and a judgment task (requiring judgments about the meaning of text), it is possible that we missed some instances that should have been ‘yes’ for this code for ‘discussing CI’. However, we sought to mitigate the difficulties associated with coding by preparing a detailed codebook, and, via blind independent coding of 30 articles, we established that our coding was reasonably reliable. Ultimately, of course, the best approach would be for other researchers to conduct close replications and replications-with-extension of our research (Brandt et al., Reference Brandt, IJzerman, Dijksterhuis, Farach, Geller, Giner-Sorolla, Grange, Perugini, Spies and Van't Veer2014; Hüffmeier et al., Reference Hüffmeier, Mazei and Schultze2016; Schmidt, Reference Schmidt2009) to establish the reliability of these and other findings from our study.

Notwithstanding those caveats, why then does it appear that JDM researchers report confidence intervals fairly frequently but almost never refer explicitly to what those intervals represent when discussing findings or presenting conclusions? One possibility is that such discussion is unnecessary because the reader can see the confidence intervals in the Results, from which they can make their own interpretations about uncertainty/precision, effect size, or patterns in the data, and then draw their own conclusions, accordingly. Certainly, one benefit of detailed and transparent reporting is allowing readers to form their own opinions about the data. That said, such an approach seems inconsistent with much of what we (as a field) do choose to discuss in our research articles because we’d be unlikely to leave-it-to-the-reader when it comes to interpreting the correlation coefficients or mean differences that are relevant to our main hypotheses. A second possibility is that confidence interval information may inform the conclusions that we draw—but rarely in ways that translate into explicit verbal expressions about range estimation or the precision of estimation. For instance, I might use the middle portion of an interval to inform what I write about the best estimate of an effect. Or, seeing that the interval includes small effect sizes might prompt me to make only cautious claims about the size of that effect. A third possibility is that journal guidelines and other best-practice prescriptions are clear and detailed on what is expected when reporting results (including expectations to report confidence intervals) but often provide little or no guidance on discussing findings and presenting conclusions. Therefore, over time, perhaps researchers learn what they should be reporting and change their practice to align with guidelines for reporting, while old habits die hard when writing the Discussion—whereby the ‘new statistics’ (Cumming, Reference Cumming2014) that they report are barely discussed. A fourth, related, possibility follows from Cohen’s (Reference Cohen1994) conjecture that confidence intervals were ‘rarely found in the literature’ because ‘they are embarrassingly large!’ (p. 1002). Perhaps, despite 30 years of encouragement to report confidence intervals in an article’s Results section, researchers are reluctant to draw attention to their ‘embarrassingly large’ intervals in the Discussion section. A fifth possibility is that reporting confidence intervals barely impacts how we think about data, if at all (Simonsohn, Reference Simonsohn2014). For example, if one treats confidence intervals as simply another procedure for rejecting a null hypothesis, one might focus only on whether the interval excludes the point null value and ignore all other information contained in the interval. If so, there would be little incentive to discuss range estimates or the precision of estimation because confidence intervals are not being used to provide information beyond that which the significance tests provide. A sixth possibility has been suggested to us, which is that, perhaps, some researchers are not confident (or not knowledgeable) about what information a confidence interval conveys. For example, Zhang et al. (Reference Zhang, Heck, Meyer, Chabris, Goldstein and Hofman2023) reported that it was common to misinterpret measures of estimation precision (e.g., SE bars) as measures of data variability, with up to half of a sample of experienced data scientists making that misinterpretation (depending upon which data visualizations were shown). Therefore, it might make sense that an author would not discuss the uncertainty or precision implied by a confidence/credibility interval if they assumed that the interval pertained to other features of their data.

Aspects of these possibilities for why confidence intervals are rarely discussed may also be relevant to the role played by effect size information within Discussion/Conclusion sections. Taking Table 3 at face value suggests reasonable regard for discussing effect size information in the Discussion or Conclusion sections of JDM research articles. In our sample, the proportion of articles that included some discussion of effect size information was slightly lower than the proportion that reported at least one standardized effect size (0.59 vs. 0.73)—a drop-off from Results to Discussion that is much less severe than we observed for confidence intervals. However, these numbers may hide just how minimal the discussion of effect size was in many of those articles. Combining 2 metaphors: we cast our net wide, and set our bar low, for what we would include as ‘discussing effect size’. First, the effects referred to could be ones that had been reported as standardized effects or as unstandardized effects (whereas for reporting results we coded only for standardized effect sizes for the reasons explained in our Method). Second, any statement about the degree, strength, or incidence rate of an effect, which went beyond dichotomous categories or mere directionality, would qualify as ‘discussing effect size’ because more than just the presence/absence of an effect was indicated. In essence, this meant that for ‘discussing effect size in words’ (the most common category of ES discussion, Table 3) it was sufficient that authors used adjectives or adverbs that could be conceptualized as anchor points on a short Likert scale (e.g., small, medium, large; e.g., slightly, moderately, substantially). To gain a fuller appreciation of the (often) limited and/or imprecise discussion of effect size information in JDM research articles, please consult Table 5 after reading the 3 sentences that follow, which describe the table’s contents. To document our decisions for the ‘discussing effect size’ code, we always sought to record 2 or more extracts for each sub-code (discussion in words, discussion in numbers). For 21 of the 88 articles coded as ‘discussing effect size’, we could only identify one such extract per article—all 21 extracts are shown in Table 5. In other words, if the 21 sentences shown in Table 5 had not been in the Results section of their respective articles, we would have identified that 45% of articles (67/150) had discussed effect size, rather than 59% of those 150 articles.Footnote ⁶

Table 5

Extracts from articles coded as having discussed effect size in words or numbers based on a single instance

Note: Row number is a reference to the master spreadsheet of our data available at osf.io/apfg8/. Emphasis added via bold-face type. Table S6 in the Supplementary Material provides additional detail on coding of discussing effect size in words or numbers, providing examples of marginal cases ultimately coded as ‘yes’ or as ‘no’.

When the data in Table 3 are seen in this light, it would appear that the Discussion sections of JDM research articles provide much more information about the presence of effects than they provide about the size of those effects. This may say more about our theorizing than our writing habits because—like much of behavioral science—our theories are often clear about the predicted direction of effects but unclear about what size of effect to expect (Meehl, Reference Meehl1967, Reference Meehl1978). Arguably, however, this is an indictment of some of the theorizing in the behavioral sciences (and a prompt to build theories which make precise predictions, for instance, through computational modeling, Guest and Martin, Reference Guest and Martin2021) not a defense for paying little heed to effect size information (Dienes, Reference Dienes2019). For example, JDM researchers interested in whether, why, and how nudges or boosts work—for which questions knowing about the presence or direction of effects might be sufficient for some scientific progress—are usually also interested in how much difference those interventions make—a determination that requires effect size information. Additionally, if effect sizes and their precision of estimation are not discussed in an article, or discussed only in general terms, this suggests that comparisons with previous research are not very detailed or precise. In the context of science as a cumulative endeavor, this may fall short of what we should be aiming for.

2.3. Abstracts are to articles as impressionism is to the art world

The scientific abstract provides the headline information about why research was done, what methods were used, what was found, and what it might mean. It’s an opportunity for authors to tell readers what’s most important about the research article. As such, the lower panel of Table 3 paints an interesting picture. Put simply, it seems that detail matters little when it comes to how results or conclusions are included in an Abstract. In the 150 papers we reviewed, the Abstract always included a statement about the presence of an effect, sometimes contained discussion of effect size (at less than half the rate for Discussion sections), but very rarely contained information about any other features that our review considered.

2.4. Exploratory analysis of reports of statistical power

Additional to our primary coding and analyses discussed above, we recorded any text extract that referred to the statistical power associated with the research reported in the article. We found that 46 of the 150 articles (0.31, 95% CI [0.24, 0.38]), reported carrying out a power analysis (none of which used a specified width of CI to report a precision power analysis). Of the articles reporting a power analysis, 0.87 (40/46, 95% CI [0.75, 0.94]) referred to an effect size in the broadest sense (e.g., ‘sufficient to detect a small effect’) and 0.72 (33/46, 95% CI [0.58, 0.83]) mentioned a specific standardized effect size (e.g., Cohen’s d of 0.5). It was interesting to us that 5 of those 33 papers (0.15, 95% CI [0.06, 0.30]), referred to a standardized effect size in the context of a power analysis but did not include a standardized effect size in their Results. Potentially, this seems at odds with the reasons why one would perform a power analysis for a specified effect size. If one had a minimum effect of interest in mind, wouldn’t one wish to compare the effect that was found against that minimum size, using the same standardized measure that had defined it? Likewise, if using an effect size from previous research for a power calculation, one might expect the ‘new’ effect to be reported in a way that allowed comparison with the prior effect (i.e., using the relevant standardized ES).

2.5. Exploratory analysis of the penetration of Bayesian methods in JDM research

Via additional coding (not preregistered) we classified 22/150 articles (0.15, 95% CI [0.10, 0.21]) as using Bayesian methods to analyze data. The methods included Bayes factors, Bayesian estimation, Bayesian regression and mixed-effects models, and various forms of Bayesian modeling and model comparison. Among the remaining 127 articles, 8 (5%) had BIC as their only Bayesian method, 8 (5%) used ‘Bayes’ only in a context unrelated the analysis of the current data, and 112 (75%) made no mention of ‘Bayes’.

2.6. Exploratory analyses of the associations between coded elements

Additionally, we examined the associations between 8 of our binary codes, primarily to explore what features of an article and its reported results predicted the discussion of effect sizes in the Discussion/Conclusions (see Table S7 and Figure S1a in the Supplementary Material). We did not attempt the equivalent analysis for discussing the presence of effects or discussing CIs because the former always occurred and the latter rarely occurred, meaning that there was little or no variance in these behaviors that could be associated with other factors. We found that discussing effect size was slightly more common in articles that (1) reported no p-value, (2) included Bayesian methods, and/or (3) were published in the 3 psychology-focus journals (vs. economic-focus journals). However, each of those association was weak and not statistically significant, with |r| $\approx$ 0.2 (Figure S1b in the Supplementary Material). Perhaps also noteworthy, neither reporting ES or reporting CIs significantly predicted the discussion of effect sizes, each with |r| < 0.1, and neither of the positive limits of the CIs for these correlations extended to r = 0.3. Thus, any association between reporting ES and discussing ES is likely weak. Also interesting to us was that reporting a power analysis was moderately associated with reporting (1) CIs or SE bars, (2) at least one p-value, and (3) standardized effect sizes in the Results (Figure S1c in the Supplementary Material). These associations were medium-sized, with r $\approx$ 0.3, and make sense in terms of how power analyses are often specified (e.g., computations within an NHST framework, predicated on a specific effect size). Also, power analyses were more frequently reported in the psychology-focus journals than the economic-focus journals.

2.7. Limitations, constraints on generality, and future directions

As discussed above, a coding exercise like the one we reported here relies on vigilance and judgment. We did our best to produce a coding scheme and protocol that promoted valid and reliable coding. Nonetheless, human fallibility dictates that some category boundaries between codes remained fuzzy and that some noise remained in the application of our coding scheme. This, together with the fact that our sample size was limited to 150 articles by our available time and resources, means that we cannot treat the proportions we report as sufficiently precise estimates to make fine-grained determinations on the basis of small differences between proportions. Therefore, we have sought only to draw conclusions that fall within the limits defined by our reliability of measurement and precision of estimation.

Hindsight revealed one limitation arising from our decision to code based on the location of information in a research article (viz. Results vs. Discussion/Conclusion vs. Abstract). First, the section headings Results and Discussion are not used in all articles; therefore, sometimes we had to decide which sections to treat as ‘Results’ and which as ‘Discussion’. These cases were mainly in the journals with an economics research orientation. Second, inevitably, and quite reasonably, academic writing varies such that some Results sections are more narrative than others. One consequence of this is that there will be instances where discussion of effect sizes took place in Results sections, which—due to its location—would not have contributed to the code ‘discussion of effect size in words’. The same could be true for textual references to confidence intervals. We maintain that it was, nonetheless, valuable to code for discussing effect sizes or confidence intervals in Discussion sections because that is, presumably, where readers expect to find the Author’s arguments about those data as pertaining to what can be concluded from the data.

There are good arguments why, often, reporting unstandardized effects in the original units of measurement is preferable to reporting standardized effect size measures (Baguley, Reference Baguley2009). Although we do not reject those arguments, we considered only standardized effect size measures for our coding of Results sections. This was because we reviewed articles covering diverse topics and using diverse methods of analysis, and did not feel we could reliably judge which (unstandardized) data summaries could be regarded as ‘suitable’ measures of effect for all articles. Therefore, our review provides a restricted snapshot of the information about effect sizes that authors present. For a more inclusive coding of effect size information that might be desirable, we think it would require a more restrictive sampling plan for what articles are considered. For example, by considering only articles that might be eligible for a meta-analysis of a specific effect, it might be easier to identify what counts as legitimate effect size information (standardized or unstandardized) for that effect (e.g., by consensus of the researchers who study that effect).

We coded for reporting and discussion at the level of the article, not at the level of each individual effect. For example, an article reporting one p-value, standardized ES, or CI, received the same code for the relevant feature as an article reporting 100 instances of that statistic. This means that we did not distinguish between an article’s primary or principal effects and its secondary or incidental effects—and to have done so for every article would, we believe, have been impossible. Nonetheless, this feature of our coding should be considered when interpreting our findings. For example, we recorded that 45% articles reported at least one confidence interval. But this cannot be interpreted as 45% compliance with guidance that ‘interval estimates should be given for any effect sizes involving principal outcomes’ (Wilkinson et al., Reference Wilkinson1999, p. 599) because it may be that CIs were only reported for effects that were secondary to the study hypotheses.

In defining our coding scheme, we were informed by longstanding recommendations to use confidence intervals and effect sizes in statistical arguments (e.g., Cohen, Reference Cohen1994; Greenwald, Reference Greenwald1975) and the guidelines that subsequently recommended reporting confidence intervals and effect sizes (e.g., APA Publication Manual, 2020; Wilkinson et al., Reference Wilkinson1999). Those guidelines imply that confidence intervals might be used alongside null hypothesis tests (Fidler, Reference Fidler2002). There are, of course, alternatives to NHST which we did not specifically include in our preregistered coding plan—most notably Bayesian methods. So, while we included Bayesian credibility intervals when coding for interval estimates, and we assumed that Bayesians would care about reporting and discussing effect sizes,Footnote ⁷ we did not code for reporting Bayes Factors or whether the information they contain is explicitly discussed in the research article. A future extension of our study might do that and/or consider whether the use of computational modeling is associated with particular patterns of reporting results and discussing findings.

Finally, we should consider whether our findings generalize from JDM research to other sub-fields of the psychological and behavioral sciences. Quantitative approaches to research are the norm in JDM, the statistical techniques employed by JDM researchers are diverse, and statistical reasoning is a core JDM topic. Therefore, we think it unlikely that JDM researchers are less focused on reporting detailed statistical information and using it to make arguments than the average behavioral scientist. For this reason, we do not think it likely that many other sub-fields have higher levels of reporting and discussing of effect sizes and confidence intervals than we have observed. Of course, it is for further research to determine whether that is the case.