1 Individual measurement

The concept of individual measurement (IM) has rightfully received a great deal of attention from generations of psychometricians and substantive researchers over the past several decades, and an informative example of the motivation behind IM is offered in the opening paragraphs of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026). An important question that would need to be asked first when interested in IM, in my view, is inquiring about the specifics of a particular intended measurement activity, which generally differ from one study to another. At the same time, there is a common feature of the pertinent measurement possesses, which is of special relevance in the overwhelming majority of empirical studies in psychology and the behavioral sciences. This is the presence of measurement error in an observed (manifest, measured) score, which is usually not negligible. I find that it is this unobserved error that contributes substantially to the challenge of IM.Footnote ²

For a given unit of analysis (ith) and measure (kth), denote the respective observed score by X_ik , the associated true score by T_ik , and the (“pure,” random) measurement error by E_ik (1 ≤ i ≤ n, 1 ≤ k ≤ p, with n ≥ 1 being group or sample size and p ≥ 1 that of the set of instrument components, questions, items, or generally measures under consideration; cf. Lord & Novick, Reference Lord and Novick1968). When information about T_ik or E_ik is desirable to obtain as part of an IM process, perhaps an immediate realization would be that there is no psychometric, statistical, or mathematical method, in my opinion, that would, in general, disentangle T_ik from E_ik in X_ik (i = 1, …, n; k = 1, …, p). This key entanglement makes IM problematic in many empirical settings.Footnote ³ One could make progress by adopting certain sets of assumptions, as often done when pursuing particular psychometric methodologies.

In the remainder of this note, in order to accomplish its aims it is posited (as is effectively the case in the discussed paper also) that (a) there are individual differences on the test score of concern (positive observed variance); and (b) the considered set of one or more test components, items, or measures are given, fixed, and the ones of interest. (With no loss of generality, person is assumed in what follows to be the unit of analysis.)

2 Classical test theory—an effectively assumption-free framework for studying behavioral measurement

In my view, there appears to be considerable misinterpretation of what classical test theory (CTT) is (and what it is not), which may be going back perhaps as far as the middle of the last century. I find that when talking about CTT or referring to it, what would be essential is to have in mind a specific, clear, and accurate definition of CTT. Fortunately, the foundation of such a definition has already been provided in the far-reaching contribution in this journal by Zimmerman (Reference Zimmerman1975). Accordingly, and as treated below, CTT is by definition not based on any assumptions (that could be falsified in contemporary behavioral research) other than the existence of the true score. The true score, which is a key concept of CTT, is defined as the expectation with respect to the propensity distribution of the observed score that is generically denoted X in the sequel and viewed as a random variable (while the corresponding true and error scores are designated T and E, with the latter being referred to as measurement error in the rest of the article; cf. Lord & Novick, Reference Lord and Novick1968). The expectation of this random variable, symbolized by ε(X) say, always exists, is well-defined, and is finite in present-day psychological research, since X is bounded both from above and below (with the absolute value of any individual realization of X being bounded from above by a finite number; e.g., Arnold, Reference Arnold1990). Therefore, the only formal assumption of CTT, viz. the true score existence, is always fulfilled in contemporary behavioral and social research, and for this reason need not be logically considered or viewed as an assumption. Hence, CTT can be treated as representing, in effect, an assumption-free framework developed to assist measurement progress in these and cognate sciences. (One can alternatively arrive at this conclusion by requiring only finite variance of X, as in Zimmerman, Reference Zimmerman1975, which is logically no assumption either, since for the above reason all observed scores in psychological research at present are associated with finite variances; I may also note in passing that the existence of variance for a random variable entails that of its mean as well; e.g., Rao, Reference Rao1973).

With this perspective, CTT is merely based on the observed score decomposition

(1) $$ \begin{align} X = T + E \end{align} $$

From the above definition of the true score it is implied that Equation (1) is (a) always true; (b) cannot be falsified (disconfirmed); (c) is not an assumption; (d) consists in its right-hand side of two uncorrelated random variables; and hence (e) does not itself represent a model (see also Footnotes 3 and 4 below). Due to these implications, in the earlier context of n persons evaluated by p measures yielding the observed score X_ik , it follows that CTT does not and need not make or adopt either or both following statements as an assumption(s): (i) the error scores of the observed measures are uncorrelated, and/or (ii) the variance of the error score E_ik is the same across persons, true score values, or time (cf. Zimmerman, Reference Zimmerman1975). For this reason, the notation σ² _ik = Var(E_ik ) would be recommended to use as a helpful symbolism when talking generally about CTT (with Var(.) denoting variance, i = 1, …, n, k = 1, …, p).

The discussion in this section relates to parts of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) as well as methods covered in their paper, and is not in agreement with several statements by them that make references to assumptions of CTT (e.g., in the opening sentence of their section on CTT, reliability, and measurement precision, as well as after their Equation (1)). These authors provide, however, no rationale for the need for such assumptions, nor do they present an indication of how one could disconfirm them if they were assumptions.

3 Model error, measurement error, and their impact on behavioral measurement

Model error (ME) is in general distinct from measurement error as well as from estimation error (with the latter depending on the used model, estimation method, and their assumptions; cf. Agresti, Reference Agresti2018). ME is a feature that typically characterizes models currently employed in empirical behavioral research, and three key properties of ME are that: (a) it may but need not include measurement error; (b) it may but need not be identical to measurement error; and (c) it “exists” before any estimation procedure is invoked and estimation error can emerge. An instructive example of ME is found in the familiar ANOVA model (whether for repeated measures or not), which is of relevance for a considerable portion of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) and is discussed further below. In that model, the ME is by definition the deviation of the response (dependent, outcome) variable from the sum of included effects–main effects and interaction(s) where applicable–irrespective of how precisely that variable is measured (e.g., Agresti, Reference Agresti2018). When the outcome variable(s) contains measurement error (that is not negligible), as one may well argue tends to be the case in most present-day psychological studies, then that ME incorporates this error while not being generally identical to it—a circumstance of relevance next.

4 Repeated measure analysis of variance with items as “repetitions”

In the section on what they refer to as “CTT methods” for evaluating conditional standard error of measurement (CStEM), Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) point to the use of repeated measure analysis of variance (RM-ANOVA), treating items as “repetitions”, which they later state is a method favored and recommended by them. Assuming they mean treating items as the repeated measures, that is, as the levels of the “time” (repeated measurement) factor in RM-ANOVA (cf. Hoyt, Reference Hoyt1941), I find that use of this statistical approach for CStEM evaluation, as outlined by them, is associated with a significant problem in general. The latter results from the fact that each of the np cells of the “design” that this ANOVA application would then be based on (with n persons and p measures), contains just one observation, viz. that of a particular person obtained at the respective measurement occasion. As a consequence, in general, the interaction term in the ANOVA model relevant then is (i) not identified (specifically, when that term is not zero), that is, not estimable uniquely; and (ii) the hypothesis of no person-by-time interaction, that is, no person-by-item interaction, is not testable (owing to 0 degrees of freedom being available then for its testing). One could make some estimation-related progress by relegating this interaction term into—that is, integrating it with—the ME of the underlying ANOVA model, which ME generally contains measurement error as part of its response variable(s) (see preceding section). With this in mind, Pfadt et al.’s reference to and use of the “interaction mean square” in their Equation (7) seems to be less than appropriate. Indeed, for the above relegation/integration reason, the right-hand side of (7) reflects the confounded effects of both (a) the ANOVA model error term per se and (b) the person-by-time interaction, while the latter is hard to consider an error term component as its mean need not be zero in general, rather than the effect of (a) only or that of (b) only.

Therefore, in my opinion, the use of the RM-ANOVA approach as outlined in Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) is, in general, difficult to see as a preferred method of CStEM estimation, and I do not share their consideration of this approach as recommendable. In particular, I do not find it to be recommendable when there is (sufficiently strong) person-by-item interaction and measurement error in the outcome variable, with this error usually being the case in empirical studies. However, whether this interaction is present or not, and, if so, how strong it is, is unfortunately not possible to examine or ascertain, owing to the above-mentioned lack of sufficient sample information needed for this process in the first place. Hence, with the above concerns in mind, the Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) utilization of the RM-ANOVA approach is, in my view, based on (i) a potentially considerably violated no-interaction assumption, which is, however, (ii) untestable. In addition, their use of this approach (iii) does not allow in general access to the measurement error variance (alone) that the CStEM critically depends on, because then (iv) the interaction-with-error-conflated term cannot be logically considered an error term of relevance for the CStEM evaluation process to begin with (see above in this section, and in particular the fact that this term in general does not have a zero mean). These important limitations cast in my opinion considerable doubts on the Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) claims (a) that this approach “does not require strong assumptions”, and (b) its general recommendation by them for CStEM estimation.

5 Alpha or reliability?

Reliability is perhaps one of the oldest concepts in psychometrics, with its earliest considerations approaching likely centenarian status. Part of the reason for this enduring interest in reliability are the facts that this coefficient is (i) the R-square index of the (conceptual) regression of true on observed score, or conversely; (ii) a non-linear function of the signal-to-noise ratio of main relevance in the engineering, technical, and other “hard” sciences; (iii) an essential part of the dis-attenuation process for evaluating true from observed correlations (e.g., Crocker & Algina, Reference Crocker and Algina2006); (iv) the correlation between two parallel measures (with uncorrelated errors), which are likely the closest one could get in psychological research to experimental replications that are of major importance for the “hard” sciences and experimental research; and hence (v) an index (on the assumptions in (iv)) of consistency of behavioral measurement, that is, repeatability of its results. These valuable properties (i) through (v) indicate, in my opinion, that the reliability coefficient has special relevance in and of itself for the behavioral and social sciences (see also Footnote 3).

Over the past eight decades or so, one of the most popular indices purporting to inform about (multiple-component measuring instrument) reliability has been coefficient alpha (e.g., Cronbach, Reference Cronbach1951, and references therein). This coefficient, however, is in general distinct already at the population level from the reliability coefficient, as has been known from a large body of research over the past 60 years or so, starting likely with the seminal paper by Novick & Lewis (Reference Novick and Lewis1967). Therefore, in the absence of specific empirical evidence to the opposite, I do not find sufficient reason for the repeated use of and/or reference to coefficient alpha by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026), including reporting and utilizing its estimates as presumably relevant for the reliability of measurement in their illustration examples. The reason is that no information (estimate) is provided by the latter authors about the discrepancy of population alpha from reliability. This omission, in addition to their not including any index of instability of the alpha estimates used throughout their illustration sections (such as confidence intervals, say), raises, in my view, significant doubts as to whether coefficient alpha, its reported estimates, and especially its substitutions for the reliability coefficient made on multiple occasions by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) could be justifiable. In particular, a main question that I see remaining with respect to their pertinent developments, is this: Are these alpha estimates and substitutions informative enough for making trustworthy judgments (even implied) or conclusions about reliability of the respective observed measures or scales (e.g., their ADD scale), which judgments and conclusions are made, used, or referred to in their data illustrations (see below)?

More specifically, I could not find a satisfactory rationale in the empirical example section(s) for the Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) decision to substitute the estimate of alpha for the reliability estimate into the formula for standard error of measurement (StEM). This is because in the absence of information in their paper regarding how far alpha may be from reliability in that study (in terms of their estimated population discrepancy), it is unknown how much these authors’ generally unjustifiable reliability-by-alpha substitution, in fact, biased the StEM that was critically used thereafter in their empirical illustration sections. (The point here is not to suggest that the StEM incorporating the reliability estimate instead of alpha’s deals away with the “average” nature of the StEM and measurement precision-related information contained in it. Rather, my point is only to indicate a potentially important bias that could be involved in their example discussions, which results from a possibly considerably mis-estimated StEM).

Similarly, I could not find a rationale provided by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) for their claim that “higher standard deviation σ_X tends to go together with increasing reliability” (in the second paragraph after their Equation (3)). This claim is hard to justify, if at all possible, in my opinion, since there is nothing that would preclude an increase in observed variance as a consequence of an increase in error variance (only), with the latter increase actually entailing diminished reliability (see, e.g., their Equation (1)). Moreover, these authors use that claim apparently in relation to a conjecture they also make in the same section, viz. that “tests of the same length but varying standard deviation σ_X and reliability ρ tend to have SEMs that show little variation” (cited also verbatim from their article). That this conjecture is no more justifiable than that earlier claim of theirs is seen by recalling that the StEM is a function of two arguments. Thereby, a couple of simple population-based examples may suffice to show the opposite of their conjecture—if observed standard deviation and reliability were 20 and .8, respectively, then the StEM results as 8.944; if they were however 100 and .4, then the StEM is 77.460, that is, more than eight times larger, which is hard to see as representing “little variation” (as stated in in that conjecture by Pfadt et al., Reference Pfadt, Molenaar, Hurks and Sijtsma2026).Footnote ⁴

6 How informative indeed is the conditional standard error of the observed test score for an empirical researcher?

The concept of CStEM, which is of fundamental importance for most of the developments in Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026), is of undoubted relevance for psychological testing. This concept is mostly utilized in their paper in direct relation to observed test scores. Yet, as is widely acknowledged, these scores are typically fallible and not unlikely to be affected by sizeable measurement error in contemporary empirical behavioral and social research. This raises the question of what the “pragmatic” value of the CStEM associated with an observed test score would be, especially for the general psychologist, as this CStEM index is what effectively most of the discussed article is built around. The importance of this question follows from the observation that measurement error adversely affects and potentially markedly the point and interval estimates of this index, by biasing the location and unduly enhancing the width of the associated confidence interval. At the same time, the resulting confidence intervals of these test score-associated CStEM indices represent essential components of the key messages aimed at by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026).

Due to this adverse measurement error effect, one may argue that, in general, there is likely to be relatively limited trait-relevant information that would be contained in the CStEM of individual test scores (and associated confidence intervals). This argument may be especially important when there are relatively large numbers of persons with given test scores, as most ADD scores reported in their Figure 4 seem to be, owing to their potentially large true score variability (and thus large error score variability) per test score. For this reason, instead of (or in addition to) being concerned with the CStEM for the observed test score, one may well submit the following point: It would likely be beneficial for an empirical researcher to use indices of precision of estimation–including respective confidence intervals–for the individual ability estimates θ_i * that one obtains from application of (correct/plausible) IRT models (i = 1, …, n); these models are relatively straightforward to fit and use these days with widely circulated software (e.g., R, Mplus, or Stata, with the first and the demo version of the second being freely available). Unlike the CStEM and confidence interval for the observed test score, which most of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) evolve around, the point and interval estimates associated with θ_i * are not similarly affected (to the same degree) by the measurement error in that score. These estimates, θ_i *, are (i) developed with the aim to be “error-free” estimates (measurement error-free, to an extent achievable with the used model and estimation method), and (ii) typically possess optimal statistical properties with respect to the underlying individual trait levels. Hence the estimates θ_i *, along with their confidence intervals, would in my opinion tend to provide more relevant information in general about what may be considered (at least conceptually) CStEM at the individual person location on the underlying latent (true) dimension of actual interest, than do the CStEM index and confidence interval associated with the fallible observed test score of nearly exclusive concern in Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026). While the latter index and interval can have empirical relevance under some and possibly restrictive conditions, depending on the research question and context, there is no apparent reason, in my opinion, why their counterpart indices for θ_i * could generally be of less importance and, in particular, provide less valuable information to empirical scientists.

Moreover, given these concerns, I would raise serious doubts regarding the potential use of the test score’s CStEM and confidence interval for making certain types of decisions with respect to studied persons. This use appears to be indicated as possible in Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026; see their first ADHD example discussion, where no statement to the opposite is found); this use and view, if acceptable by them, is, however, what I would disagree with. Rather, I would recommend abstaining from utilizing the CStEM and confidence interval associated with the observed test score. Instead, expert substantive considerations and deliberations should at least be an essential part of the process, possibly leading to this kind of decisions, if there are sufficiently strong reasons for making them in the first instance.

Based on the preceding discussion in this note, I find that the pertinent developments and arguments in Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) are similarly not convincing in their primary focus on the observed test score throughout most of their article. This opinion of mine can, in addition, be seen as resulting from the following four observations. One, two persons with substantively different patterns of responses on a set of (unidimensional) test/scale components (questions, items, problems, tasks) may still receive the same observed test score, in addition to possibly being associated with measurement errors with considerably varying magnitude (in absolute value) and opposite signs. Two, this test score is, in general, not a sufficient statistic for the underlying individual trait level. Three, all persons with the same test score, regardless of their number, which could actually be large, still get the same CStEM with each of a number of methods referred to or used for its evaluation in the discussed paper (as is also seen for every method reported in their Figure 4). Four, Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) appear to be silent on whether any of the discussed methods for CStEM evaluation (a) needs to assume the same error variance for all persons having the same (given) observed score (with their number being potentially very large); or (b) reflects otherwise only the average of these error variances in the resulting CStEM associated with that (given) test score. On this same matter, there seems to be no statement or evidence provided in their paper that disconfirms either of the statements in (a) and (b) (in the sentence preceding the present one). In summary, in my opinion, the limitations pointed out in this section of the note seem not possible to be counteracted effectively and to a satisfactory degree by the CStEM associated with the observed test score, at least not to an extent warranting general recommendation of this CStEM as apparently advanced in the discussed paper.

7 Tutorial guidance for empirical researchers involved in behavioral testing

While the list of recommendations that Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) provide toward the end of their article contains useful guidelines, behavioral measurement is often richer in aspects that need to be attended to than a relatively limited list like that could point to, if to be optimally relevant. Human behavior measurement, in my view, tends to be associated with a multitude of idiosyncrasies when considered across various empirical settings, while many or most (if not all) idiosyncrasies need to be adequately dealt with in this activity. The underlying reason is that by its very nature, IM is ultimately a potentially greatly challenging process, in that it is inherently interested in missing values for all studied units of analysis, namely the individual studied ability or trait values. Thereby, these values are not only unavailable/missing but in principle could have any magnitude (and possibly sign; see also Footnotes 2 and 3). Instead of such a list of recommendations, therefore, I would like to argue for pointing out the relevant aspects, sides, and facets of a behavioral measurement endeavor under consideration, with that activity likely depending on the subject-matter domain and research question(s) pursued.

In this connection, I find that the recommendation list offered at the end of the main text of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) misses emphasis on some relevant, in my view, initial activities that, to a degree, were alluded to earlier in the present note. Specifically, I would argue for an essential recommendation to examine the validity (plausibility) of any model used in pursuing IM and/or related matters like precision of estimation, by accounting, if need be, for possible (i) clustering effects as well as (ii) substantial unobserved heterogeneity. This is because the presence of (i) and (ii) in empirical research has been on the rise in recent decades, owing to increasingly complex data collection procedures and growing population heterogeneity (cf. Raykov, Reference Raykov and Hoyle2023). Thereby, while engaged in fit evaluation of an used model, several issues need to be kept in mind: (a) special attention should be paid to examining each model and estimation method assumption (including conditional independence/uncorrelatedness and possibly additional parameter restrictions); (b) no further assumption(s) should be made while examining a given assumption(s); and (c) no unnecessary or implausible (incorrect) assumption should be part of an utilized point or interval estimation procedure. Such an assumption, which is better to avoid, is, for instance, that of normally distributed error scores when dealing with dichotomous or polytomous items, or with test scores resulting from a limited number of items. The latter possibly implausible assumption is made by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) in their illustration example that is based on nine items. Hence, one may ask to what extent their conclusions regarding the performance of the StEM and CStEM indices and confidence intervals for the ADD test scores of interest may have been adversely affected by this potentially considerably violated normal errors assumption. Furthermore, in order to utilize what is referred to as IRT-based CStEM in their example, these authors a priori assume unidimensionality of the utilized nine ADD items without evidence of this assumption being plausible or tested. While this omission does not (automatically) prove their empirical results as wrong, it casts considerable doubts on their proposed interpretation of at least part of their findings (specifically those obtained under that untested item unidimensionality).

Moreover, I find that Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) hardly offer effective advice for dealing with settings where unidimensionality does not hold, or pertaining to the exploration of a test’s underlying latent structure that can and often will be of particular importance in psychological testing. This is because much of the more involved and frequently needed work relevant to behavior evaluation may be seen as beginning under such circumstances. The underlying reason is that, possibly due to validity-related reasons, a test or scale of interest is not unlikely to be multidimensional rather than unidimensional (at least initially), with its latent structure, in addition, depending on the population being studied (e.g., Raykov, Reference Raykov and Hoyle2023, and references therein). Further, more attention may also need to be paid to the possibility of (some) bi-factor models being plausible, a circumstance partly indicated in passing by Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026). The relevance of bi-factor models in this connection results from the fact that, according to a key feature of such models, it is generally not possible to find a unidimensional subscale or testlet from a test or scale complying with a bi-factor model, whereas it would be generally incorrect to expect that its “distance” from unidimensionality would be sufficiently small to ignore. Since most of the discussion of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) assumes unidimensionality, a reader of a tutorial would also want to see some guidance on how to handle a test score resulting from a measuring instrument that is not unidimensional. Unfortunately, such guidance is not provided by them, and the pertinent discussion seems at times somewhat informal and developed in less than efficient and broad terms.

More generally, the recommendation list of Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026) appears less than concrete if not largely silent on the clustering, population heterogeneity, and violation of unidimensionality settings, as well as recommendable actions to deal with them. In my view, these settings are of high importance to a large, if not substantial, part of contemporary behavioral and social science research. Last but not least, I have found that the list and relevant discussion sections are limited in effective advice for choosing between the many CStEM evaluation and related methods mentioned, pointed to, or referred to in Pfadt et al. (Reference Pfadt, Molenaar, Hurks and Sijtsma2026). This is, in my opinion, a significant limitation of their tutorial, owing especially to the fact that most of these methods can yield what seems to be considerably different results in terms of CStEM as evinced by the reported indices in their Figure 4.