1 User-friendly software

Over the past decades, rapid advances in psychometrics—which, in my view, primarily serves psychology and education—have widened the gap between psychometrics and these fields, making dissemination one of its most important tasks. So, I welcome the accessible tutorial in Psychometrika on the conditional standard error of measurement (CSEM) by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026), and the CSEM estimates provided in JASP. Nowadays, most authors implement newly developed psychometric methods in R or Python, which may suggest that advocating their implementation in user-friendly software is no longer necessary. However, I disagree and argue that the notion of “user-friendly software” requires further qualification, and that making psychometric methods available in software such as JASP, alongside R and Python, merits continued support.

The majority of users of psychometric methods—particularly those outside academia—are unfamiliar with R and Python. In my institution, approximately 500 students graduate each year, of whom only about 30—those pursuing academic careers—have strong R skills. The remaining 470 follow content-oriented programs, often including clinical training, leaving no time to become proficient in R. Yet, many will apply psychometrics in practice. They will assess children in forensic youth care based on test results, monitor the progress of special-needs students, or conduct small-scale studies within the organization they work. For psychometric methods to have a societal impact, it is not sufficient that software is user-friendly for psychometricians, it must also be user-friendly for practitioners.

JASP may be on its way to becoming user-friendly for practitioners (see also Shepard & Richardson, Reference Shepard and Richardson2024). Based on our institution’s recent transition from SPSS to JASP, I learned that students find JASP intuitive. JASP also offers three key advantages over SPSS. First, as free software, JASP remains accessible after graduation, unlike SPSS, whose high licensing costs often prevent continued use. Second, JASP keeps pace with developments in psychometrics, as illustrated by the availability of CSEMs. Third, JASP provides R functions that enable further investigation of its methods through simulation studies, potentially improving the software or stimulating new research.

2 Bias and sampling variability in estimated CSEMs

In their Figure 2, Pfadt et al. illustrated the use of the unconditional standard error of measurement (SEM) and CSEMs. Its key message, in my view, is that individual test scores should be accompanied by error bars. However, whether these error bars should be based on CSEMs estimated using the ANOVA method (from here on referred to as ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ ), as advocated by Pfadt et al., remains open to debate, because bias and variance in CSEM estimates are not yet well understood and may obscure the additional accuracy they promise.

The bias of the estimated SEM in a sample ( $\widehat{\mathrm{SEM}}$ ) can be approximated by

(1) $$\begin{align}\mathrm{Bias}\left[\widehat{\mathrm{SEM}}\right]\approx -\frac{\unicode{x3c3}_X}{2\sqrt{1-\unicode{x3c1}}}\;\mathrm{Bias}\left[\widehat{\unicode{x3c1}}\right]\end{align}$$

(Van der Ark et al., Reference Van der Ark, Emons, Ellis and Sijtsmain press). As test-score standard deviation ${\unicode{x3c3}}_X$ serves as a scaling parameter, $\mathrm{Bias}\left[\widehat{\mathrm{SEM}}\right]$ depends on reliability $\unicode{x3c1}$ and the bias of its sample estimate $\mathrm{Bias}\left[\widehat{\unicode{x3c1}}\right]$ . For lower bound reliability estimates, such as Cronbach’s alpha, $\mathrm{Bias}\left[\widehat{\mathrm{SEM}}\right]$ is positive, yielding conservative SEM estimates. For example, with ${\unicode{x3c3}}_X=1$ , $\unicode{x3c1} =0.9$ and $\mathrm{Bias}\left[\widehat{\unicode{x3c1}}\right]=-0.05$ , we obtain $\mathrm{SEM}=1\sqrt{1-0.9}\approx 0.316$ , whereas $\mathbb{E}\left[\widehat{\mathrm{SEM}}\right]\approx -\frac{1}{2\sqrt{1-0.9}}\left(-0.05\right)\approx 0.395.$ As CSEM estimates also depend on $\unicode{x3c1}$ and $\mathrm{Bias}\left[\widehat{\unicode{x3c1}}\right]$ , they are likely to be biased as well.

In addition to bias in the reliability estimator, binning (i.e., combining adjacent test-score groups to obtain sufficient sample sizes per group) may introduce additional bias. Binning is generally required in the ANOVA, Feldt, and Thorndike methods. Pfadt et al. recommended using groups of at least 25 respondents; in JASP, the argument caseMin controls the minimum group size. I examined the effect of caseMin on CSEM estimates using the ADD data—a dataset available in JASP containing the scores of 1,162 respondents on nine polytomous items, used by Pfadt et al. to illustrate estimated SEM and CSEMs. Figure 1 shows that caseMin affects both the absolute magnitude of ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ and its relative ordering across test scores. This was also observed for the Feldt and Thorndike methods (not shown). Furthermore, for the ANOVA method setting caseMin to its maximum did not yield the estimated SEM, whereas it did for the Feldt method.

Figure 1

The estimated CSEM using the ANOVA method for test scores 6 (blue, smallest linewidth), 18 (red), and 34 (black, largest linewidth) on the ADD-data as a function of the minimum sample size used for binning (caseMin) expressed on a log2-scale. The dashed horizontal line represents the estimated SEM and the vertical dotted line represents caseMin = 25.

Figure 1 Long description

A line graph with the horizontal x-axis labeled case Min on a log 2 scale with tick marks at 10, 16, 32, 64, 128, 256, 512, and 1024. The vertical y-axis is labeled C S E M with values ranging from 1.8 to 3.0.

Three step-like lines represent different test scores:

* A thin blue line for score 6 starts at 1.95, remains flat until case Min 32, then fluctuates before rising sharply toward 3.1 at the far right.

* A medium red line for score 18 starts at 2.4, drops to 2.2 near case Min 40, and then rises steadily after case Min 200.

* A thick black line for score 34 starts at 2.5, fluctuates between 2.4 and 2.7, and eventually merges with the red line at the highest case Min values.

Two reference lines intersect the data:

* A horizontal dashed line at C S E M equals approximately 2.18 representing the estimated S E M.

* A vertical dotted line at case Min equals 25.

Under a multinomial sampling scheme, Van der Ark et al. (Reference Van der Ark, Emons, Ellis and Sijtsmain press) derived a standard error for $\widehat{\mathrm{SEM}}$ , and simulations showed that these standard errors are relatively small for $N>200$ . For the ADD data, $\widehat{\mathrm{SEM}}=2.182$ , $\mathrm{SE}\left(\widehat{\mathrm{SEM}}\right)\approx 0.024$ , and the relative standard error— $\widehat{\mathrm{SEM}}/\mathrm{SE}\left(\widehat{\mathrm{SEM}}\right)$ —is approximately 1.1%, indicating that sampling variability has little effect on $\widehat{\mathrm{SEM}}$ . However, the effect of sampling variability on estimated CSEMs is yet unknown.

In a short simulation study, I examined whether using ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ , as recommended by Pfadt et al., improves individual-level decisions by comparing the coverage rates of 95% confidence intervals based on (i) the estimated SEM and (ii) ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ . To mimic realistic psychological test data, a three-dimensional graded response model estimated from the ADD data using the R package mirt (Chalmers, Reference Chalmers2012) was used as the data-generating model, which was the best fitting model using the BIC as a criterion. For seven true-score values, computed from the data-generating model (see Table 1), coverage rates based on 10,000 replications were computed using JASP’s R functions for SEM and CSEM estimation. Details of the true-score computation and accompanying R code of the simulation study are available on Open Science Framework at https://osf.io/xyfsp.

Table 1

Coverages rates for the 95% confidence intervals based on $\widehat{\mathrm{SEM}}$ and ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ , for seven true-score values

Table 1 Long description

The table consists of a header row and two data rows.

Header Row: The first column is labeled Estimator. The subsequent seven columns are labeled under the heading True score with the following values: 0.1152, 0.6945, 3.6952, 11.2736, 19.2848, 26.6288, and 31.6043.

Row 1: Estimator S E M hat. The coverage rates are 1.000 (bold), 0.999 (bold), 0.979 (bold), 0.945, 0.945, 0.966 (bold), and 0.988 (bold).

Row 2: Estimator C S E M hat sub A N O V A. The coverage rates are 0.592 (bold), 0.750 (bold), 0.898 (bold), 0.949, 0.967 (bold), 0.973 (bold), and 0.989 (bold).

Note: Bold values indicate coverage rates falling outside the 95 percent Agresti-Coull confidence interval based on 10,000 replications.

Note: T, true score; $\widehat{\mathrm{SEM}}$ , estimated standard error of measurement; ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ , conditional standard errors estimated using the ANOVA-method. Coverage rates in bold fall outside the 95% Agresti–Coull confidence interval. Coverage rates are based on 10,000 replications.

Except for two true scores, confidence intervals based on the $\widehat{\mathrm{SEM}}$ showed overcoverage, which means these confidence intervals were too large—they contained the true score more often than claimed. This can be expected because $\widehat{\mathrm{SEM}}$ is positively biased (too large) if reliability is underestimated (Equation (1)). For the four highest true-scores, the confidence intervals based on ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ showed coverage rates similar to those of the $\widehat{\mathrm{SEM}}$ but severe undercoverage for the three lowest true scores, which means that the confidence intervals were too optimistic—they contained the true score less often than claimed. Hence, in this study, $\widehat{\mathrm{SEM}}$ provided better coverage than ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ , which is not in line with the suggestion that CSEMs are more recommended. The simulation is too small to draw any conclusions about the appropriateness of using ${\widehat{\mathrm{CSEM}}}_{\mathrm{ANOVA}}$ , but I hope its results and the considerations with respect to bias and variance may provide directions for future research.