3.1. Diagnostic validity

Validity in the DSM, referred to as diagnostic validity, was first articulated by Eli Robins (1921–1994) and Samuel Guze (1925–1996) in 1970 and subsequently updated alongside the various iterations of the DSM’s categorical classification system in the context of clinical medicine. Robins and Guze were motivated by an interest in achieving psychiatrist Emil Kraepelin’s (1856–1926) big idea: psychiatric disorders may be shown to be discrete disease entities that can be accurately identified through clinical observation of their signs and symptoms, direct observation of their pathological anatomy and underlying physiology, or through the study of their etiology. Their overall strategy was simple: (i) identify discrete and homogenous diagnostic groups based on what we know, being observable signs and symptoms, and (ii) study those diagnostic groupings to discover their underlying and corresponding pathological processes.

Diagnostic validity is the extent a DSM-based diagnostic category (e.g., Major Depressive Disorder), comprised of a set of operationalized diagnostic criteria being observable signs and symptoms considered infallible indicators of an underlying clinical syndrome, is supported by a specific set of validators, understood as acceptable sources of validating evidence for a diagnostic category. The central question is “whether we have any confidence in the validity of this syndrome based on the set of validators” (Kendler et al. Reference Kendler, David Kupfer, Phillips and Fawcett2009, 8). Diagnostic validity is based in evaluations made by expert-led committees from twenty-one disorder groupings of the “strength of evidence for each of the validators” and “the overall strength of evidence across all validators” (Kendler et al. Reference Kendler, David Kupfer, Phillips and Fawcett2009, 3).

Evaluations of the accumulation of evidence from the validators, i.e., the aggregation of the validators, contribute to a judgment regarding the diagnostic validity of a diagnostic category. An implicit hierarchy among the validators exists, which reflects Kraepelin’s original clinical method of prioritizing predictive sources of validating evidence, such as course of illness (i.e., evidence of a diagnostic category’s predicted duration) and response to treatment, over and above external sources of validating evidence such as biological markers (i.e., evidence describing the pathophysiological correlates)—although evidence from biological markers has recently increased in priority within the past 15 years. Table 1 lists the current set of validators for the DSM, of which those denoted by an asterisk (*) are deemed high priority.

Furthermore, whether a diagnostic category is judged to have sufficient diagnostic validity is only one of several considerations for the category’s placement in the DSM. A host of other considerations includes evidence the diagnostic category sufficiently meets criteria for a “Mental (Psychiatric) Diagnosis”; evidence of the category’s reliability, defined by DSM-5-TR as “the degree to which two clinicians could independently arrive at the same diagnosis for a given patient” (American Psychological Association 2022, 8); and clinical utility, defined as the ability to “help clinicians to determine prognosis, treatment plans, and potential treatment outcomes for their patients” (American Psychological Association 2022, 14); as well as many practical utility-related considerations such as the need for the category, ease of use, clinician acceptance, any potential harm caused by the category, and evidence of available treatments.

3.2. Structure-first psychometric validity

HiTOP’s specific conception of validity, which I interpret as structure-first psychometric validity, developed independently from diagnostic validity in the context of psychometrics, a quantitative scientific research tradition traced to the introduction of the common factor model of general intelligence by Charles Spearman in the early twentieth century. HiTOP’s psychometric approach rejects the DSM’s notion of a psychiatric disorder being the result of an underlying discrete clinical syndrome and instead views psychopathology as arising from coherent and distinct latent (unobserved) dimensions as represented by its HiTOP constructs that hold a shared causal influence on a set of indicator (observed) variables, being the symptom groupings of psychopathology. For example, under HiTOP, a clinician would not interpret the symptoms of depression as arising from a single underlying syndrome such as Major Depressive Disorder but would instead view depression symptoms as interrelated and overlapping with one another to varying degrees, such that there is ultimately a broader and coherent dimension, the internalizing spectra, which underlies all symptoms of depression and symptoms from related disorders (e.g., generalized anxiety).

HiTOP’s data-driven strategy, described as evidence-over-experts, uses a statistical technique called factor analysis to derive the most common hierarchically organized factors of psychopathology, i.e., HiTOP spectra, which it refers to as the structure of psychopathology, from psychological testing data (Kotov et al. Reference Kotov, Krueger, David Watson, Achenbach, Althoff, Bagby and Brown2017). HiTOP then organizes the empirically derived structure from broadest (e.g., the general factor of psychopathology or “p-factor”) to narrowest (e.g., homogeneous symptom components/maladaptive traits). HiTOP ultimately seeks to validate the entire data-driven structure to inform a future transdiagnostic classification system that may replace the DSM (Forbes et al. Reference Forbes, Ringwald, Allen, Cicero, DeYoung and Eaton2024).

Structure-first psychometric validity falls within psychometric validity, broadly understood as the degree to which a test, being a response to a standardized situation devised to measure a psychological construct, has the desired psychometric features (e.g., various conceptions of validity, reliability, utility, etc.). Unlike diagnostic validity, structure-first psychometric validity is based in construct validity, the unified conception of validity that encompasses all psychometric validity within psychological testing. Structure-first psychometric validity is centered on supporting the development of psychiatric classifications that carry both scientific accuracy, understood as the degree HiTOP’s constructs represent true features of psychopathology, and clinical utility, the degree classifications are considered practically and pragmatically clinically useful.

Structure-first psychometric validity is an ongoing three-stage validation process. The first and most crucial stage is the evaluation of structural validity (hence, structure-first), being the degree to which a particular construct accounts for the empirically observed covariance (i.e., the direct relationship) between different signs and symptoms of psychopathology. Structural evidence is based in exploratory and confirmatory factor-analytic research with a preference for continuous latent variable models. Such models produce factor loadings, i.e., standardized correlations between the original variables and an underlying factor (the construct), which are assessed based on the model’s goodness of fit. The second stage is an evaluation of external validity, referring to the degree to which evidence for a certain HiTOP construct correlates with other (relevant) indicators of that construct. The third stage is an evaluation of reliability, the extent to which all items on a test measure the same construct, as well as clinical utility, and predictive utility, the degree to which a construct is helpful in differentially predicting outcomes of interest. Through structure-first psychometric validity, HiTOP aims to validate its constructs as well as the broader system of multiple, hierarchically organized constructs, i.e., the entire HiTOP hierarchical model itself.

3.3. Network psychometric validity

Network psychometric validity, also based in psychometric validity, operates within the network perspective, which represents a family of models and methodologies that draw on the network approach to psychopathology, network theory, complex systems theory, and applications in network science to model and study mental disorders as complex causal systems. For example, under the network approach, a depressive episode is hypothesized to arise from the causal interaction between symptoms such as depressed mood, anhedonia, and others (e.g., insomnia, fatigue). As a result, symptoms are not conceived as fallible indicators of some underlying common cause such as the DSM’s Major Depressive Disorder or HiTOP’s Internalizing Spectra. Instead, it is the mutual interaction between symptoms that constitutes depression itself.

Unlike the DSM or HiTOP, the network approach to psychopathology, which first gained traction from psychometrician Denny Borsboom (Reference Borsboom2008; Reference Borsboom2017) and affiliated members of a psychometrics research group based in the Netherlands, is not affiliated with any single governing body or research organization, nor does it (yet) directly inform a system of psychiatric classification. The aim of the network approach is to replace the data-driven latent variable models typical of the HiTOP approach, which the network approach claims do not support testable explanations, with theory-based network models of psychopathology. This “theory-first” approach means selecting network models based on theoretical reasons which, it is argued, provide the necessary rationale for developing and testing network theories to inform future psychiatric classifications.

A network model of mental disorders is a statistical model that represents features of a mental disorder as derived from a specific network hypothesis, being a testable and falsifiable hypothesis of how the components in a network influence each other over time. A network is a representation of the relationships (formally called edges) between constituent variables (formally called nodes) within a system. In psychopathology networks, the nodes represent various constituent elements of psychopathology (e.g., symptoms, biomarkers, cognitive processes). Edges represent conditional associations between nodes. Two or more networks may be connected by what has been referred to as bridge symptoms (or bridge nodes). The external field is an area outside of the network whose components (e.g., “stress”) may causally intervene on the nodes or edges inside the network. In a network model, the more thorough the connections of the nodes in the network, the more likely the network is to remain in a dysfunctional state even after removal of the original biopsychosocial variables (i.e., hysteresis), and may also reflect higher levels of severity in dysfunction.

Network psychometric validity can be summarized as being comprised of three stages. The first stage is the validation of the individual components in the network, based in the concept of node validity, a model-specific validation process (Bringmann et al. Reference Bringmann, Albers, Bockting, Borsboom, Ceulemans, Cramer, Epskamp, Eronen, Hamaker, Kuppens, Lutz, McNally, Molenaar, Tio, Voelke and Wichers2022). Node validity is a two-step validation procedure that involves (i) node selection, referring to the adequacy of selecting appropriate variables as nodes in a network model, and (ii) node assessment, referring to the quality of the operationalizations used for selected variables. The second stage is the validation of the dynamical relations between the components, i.e., the validation of the network structure, whereby the dynamic relation between specific nodes may be understood as a kind of useful construct to be validated. A third and final stage in network validation is to go beyond validating nodes and their relations, and toward deriving empirical implications from the model to test (and subsequently validate) a network hypothesis. At present, this stage is not discussed in terms of an explicit validation procedure, but rather in terms of validity-adjacent concepts such as testability, the ability to subject a hypothesis to appropriate empirical testing conditions, and falsifiability, the ability of a hypothesis to be rejected by empirical testing.

3.4. Etio-pathophysiological validity

Etio-pathophysiological validity is the conception of validity of RDoC, an alternative research framework of the National Institute of Mental Health (NIMH). Initially launched in 2009, in 2015 the RDoC program changed leadership following Thomas Insel’s departure as director, resulting in a significant change in research priorities—an RDoC 2.0—that would eventually lead to the retirement of the original RDoC matrix. RDoC represents an integrative approach to validation that attempts to bring the syndromal model, psychometrics, dimensionality, and multicausality together with cognitive neuroscience in one unifying validity framework. RDoC’s overall strategy is to group patients for clinical studies based on fundamental dimensions of behavior and neurobiological measures (genes, circuits, etc.). By adopting an approach that breaks up the DSM’s diagnostic categories like HiTOP and the network approach, yet unlike those approaches fixes measuring the underlying biology of the phenomena as its starting point, RDoC aims to inform a more valid classification system in the future based firmly in biology.

The RDoC approach employs what I interpret as a dual-track model of validation. The first track, which I term biology-first function validity, is centered on the validation of the tools of the RDoC Framework, referred to as the “concepts for investigation,” with a primary focus on RDoC constructs. These concepts include six major functional domains, which represent current understanding of the major systems of cognition, motivation, and social behavior, i.e., those systems which, when there is dysregulation and dysfunction within/across them, are thought to give rise to psychological and behavioral impairments. Each domain is accompanied by three to six constructs, i.e., concepts summarizing data about a specified psychological/biological dimension of behavior, recently defined as empirical functions. Units of analysis are the methods and instruments used to study the constructs from a normal to abnormal range of functioning.

The second track, which is intended to be informed and supported by the first, is the open-ended and under-specified validation of a future diagnostic classification system, which I term biology-first syndromal validity. Biology-first syndromal validity would be applied to a classification system that can accommodate RDoC’s specific hypothesis concerning mental disorders as representing broad and biologically heterogeneous syndromes as opposed to discrete clinical syndromes and would require validation using etiology, pathophysiology, prognosis, and treatment response measures. A validation process on which biology-first syndromal validity may come to be modeled comes from an RDoC predecessor, the Bipolar and Schizophrenia Network for Intermediate Phenotypes (B-SNIP). B-SNIP’s four-stage process for developing valid psychiatric classifications outlined by Keshavan et al. (Reference Keshavan, Clementz, Pearlson, Sweeney and Tamminga2013) amounts to (i) agnostic deconstruction of disease dimensions, identifying disease markers and endophenotypes; (ii) applying such markers across translational domains from behaviors to molecules; (iii) re-clustering cross-cutting bio-behavioral data into biotypes, being transdiagnostic clusters defined by responses on measures across units of analyses using modern phenotypic and biometric approaches; and (iv) validating biotypes using etio-pathology, outcomes, and treatment-response measures.

The term biology-first applies to both tracks of validation and is in service of RDoC’s primary aims: (i) develop an etiological and pathophysiological understanding of human systems of normal and abnormal functioning; and (ii) contribute to a future biologically based system of classification. In the first track, biology-first is evidenced by the specific criteria for RDoC constructs. For an RDoC construct to be initially selected and subsequently considered valid, it must include evidence that a neural circuit or biologically based system plays a role in implementing the function. In the second track, biology-first reflects the notion that a future classification system will be validating syndromes that have been shaped via an understanding of their biological basis. As mentioned, a leading frontrunner is the new classificatory concept of biotypes, which RDoC considers to be “more biologically valid groupings than the diagnostic categories” (Cuthbert Reference Cuthbert2020, 84).

4.1. A return to the original validators of Robins and Guze (Reference Robins and Guze1970)

One of the most surprising commonalities is that, for those frameworks seeking to inform a future psychiatric classification system, i.e., HiTOP and RDoC, there is a return to some of the original validators of Robins and Guze (Reference Robins and Guze1970) such as etiology (i.e., family history), prognosis (i.e., the likely course or outcome of the illness), and treatment response. While the DSM, RDoC, and HiTOP have all emerged within the context of patient care and thus a focus on specific validators which support the clinical utility of psychiatric diagnoses may seem expected, a return to Robins and Guze is unforeseen for two reasons. First, the primary motivation of the alternative frameworks was to go beyond the DSM’s iterative approach to validation and toward a “paradigm shift model” (Kendler and First, Reference Kendler and First2010) where the original approach is discarded due to the belief that the limits within the original paradigm had been reached. Second, both HiTOP and RDoC have been so outwardly critical of the DSM’s standards for validity as well as utility that a return to what they deem problematic seems counterintuitive.

To clarify, HiTOP and RDoC do explore other validity avenues first. HiTOP first develops structural evidence of the HiTOP spectra, and then proceeds to validate HiTOP constructs against some of the original validators. For RDoC, following a focus on biology-first functional validity of its RDoC constructs, validation of a future RDoC-informed classification system via biology-first syndromal validity then depends on prognosis and treatment response. Ultimately, a return to some of the original validators makes HiTOP and RDoC much more in line with the DSM’s iterative approach in comparison to the paradigm shift models they have previously been characterized under.

4.2. Expert curation

A second unpredicted commonality is that each alternative framework employs expert curation, meaning decisions as to what is ultimately included in their model(s) or classification system(s) are based on compromises between experts. These experts, specifically MD and PhD key opinion leaders who are selected to an evaluation or oversight committee, are tasked with assessing and judging the validating evidence in relation to other various epistemic and non-epistemic aims of an approach, e.g., compromising between the truth of the classification and the degree to which it is considered to be useful in clinical practice. This is also surprising since the primary criticisms of the DSM are on the one hand that it is too expert-based and is thus overtly biased and subjective, and on the other hand that it fails to incorporate the “right” experts by not including patient perspectives in the revision process (e.g., see Tekin Reference Tekin2022; Knox Reference Knox2022). When alternative frameworks are promoted, the perception they put forward is that their “data-driven,” “theory-driven,” or “biologically driven” approach removes the overreliance on experts, permitting an empirically based process that better “discovers” its classifications in an “objective” and “scientific” manner.

To their credit, all the frameworks are trying to improve their approach to expert curation. The DSM seeks to systematize and standardize its expert-driven decisions in a way that permits empirical evidence to carry more weight. HiTOP distinguishes various features in its overarching model, such as the separation between Somatic and Internalizing Spectra, or the “p-factor” based on expert decisions as opposed to those features being empirically determined by factor analysis. At the same time, HiTOP implements an evidence-based GRADE rating system in their revision decisions to make the use of experts more objective. The Network Approach utilizes a partially expert-led process of node selection and node assessment, whereby the “clinical or theoretical hypothesis of a clinician … plays a role in the choice of the set of variables or nodes in the network” (Bringmann et al. Reference Bringmann, Albers, Bockting, Borsboom, Ceulemans, Cramer, Epskamp, Eronen, Hamaker, Kuppens, Lutz, McNally, Molenaar, Tio, Voelke and Wichers2022, 3). In turn, they maintain that rigorous empirical testing is the ultimate arbiter. RDoC developed the original RDoC matrix during a series of two-day workshops whereby experts hurriedly populated RDoC constructs into the matrix they took to be the most reasonable. RDoC still adopts specific tasks and experimental paradigms experts deem important, but their strategy now considers constructs in the updated RDoC Framework as more exemplars to be tested, thus reducing the strength of their claims regarding expert-selected constructs. Notably, the focus across frameworks on making expert decisions more “objective” advocates for what Gagné-Julien has criticized as the “ideal of value-free science” (Reference Gagné-Julien2021, 9401) as well as a narrow conception of who counts as a so-called expert.

4.3. Validity is broadly understood as that which is considered “good” or “desirable”

Despite the elaborate presentation of validity as being empirically or scientifically based, accompanied by long lists of validators or intricate sequencing of how such and such evidence should be evaluated, a third commonality is that validity for each approach in the broadest conception boils down to that which is considered to be “good” or “desirable.” That validity in psychiatry may be interpreted so broadly is not inherently a criticism. In the 2010s, the great validity debate within educational and psychological testing centered specifically on how and whether to define validity in a narrow or very broad sense (Newton and Baird Reference Newton and Baird2016). The question was whether validity should be defined in a narrow, traditional conception of determining whether a test is actually measuring the thing you want to measure (Markus Reference Markus2015)—which is far more difficult to establish—versus a broad conception so that validity is more flexible and may come to mean anything to do with whether an assessment procedure is “good” or “bad” (Newton and Shaw Reference Newton and Shaw2015). How to define validity was essentially a debate concerning how the term validity should be used, while the more difficult philosophical questions as to whether validity should establish a link between the construct one is attempting to measure and the measuring instrument, or whether validity should be thought of as demonstrating the thing being measured exists, did not drive the debates.

Evidence to suggest that validity may be understood more broadly in psychiatry is that no current conception of validity across the four approaches engages in more difficult measurement questions in terms of what we are really doing when we’re measuring and subsequently validating, or what is really required of psychological measurement so we may be confident the thing underlying the construct (e.g., a clinical syndrome, the general factor of psychopathology) exists. There is a heightened focus on conceptualizing the phenomenon, thinking about how to study it, deciding which types of evidence are important, and developing and testing specific hypotheses, but psychiatry has seldom engaged with the more challenging and rigorous philosophical problems of measurement.

5.1. Difficulties evaluating between frameworks

One of the most immediate implications is the difficulty in comparing frameworks in terms of which is more valid. Consider a recent debate between critics of HiTOP and HiTOP proponents that represents just a microcosm of the ongoing validity debates shaping validity standards. In their paper “Folk Classification and Factor Rotations: Whales, Sharks, and the Problems with the Hierarchical Taxonomy of Psychopathology (HiTOP),” psychologists Haeffel et al. (Reference Haeffel, Jeronimus, Kaiser, Soyster, Fisher, Vargas, Goodson and Lu2021) challenged the notion that HiTOP significantly improves upon the DSM. In critiquing HiTOP, they point to two major weaknesses that limit its potential toward achieving validity: its “data-driven” approach is atheoretical and, as a result, is unfalsifiable, meaning it is not suitable for theory-building. They further criticize HiTOP’s simple-structure factor-analytic approach and its use of the degree of model fit as an indicator of validity (i.e., structural validity), claiming “… HiTOP’s hierarchical approach is not valid” (2021, 262) nor does it hold the potential for achieving validity.

In their response, “Answering Questions about the Hierarchical Taxonomy of Psychopathology (HiTOP): Analogies to Whales and Sharks Miss the Boat,” HiTOP proponents DeYoung et al. clarify HiTOP’s validation procedure, arguing its atheoretical, data-driven approach “maximizes coherence of constructs and distinctiveness between them” (2021, 280) and is thus beneficial and valid, while also suggesting their approach is capable of hypothesis testing “according to their fit to the data” (2021, 281). In response to the response, Haeffel et al. assert DeYoung et al. (Reference DeYoung, Kotov, Krueger, Cicero, Conway, Eaton, Forbes, Hallquist, Jonas, Latzman, Ruggero, Simms, Waldman, Waszczuk, Widiger and Wright2021) fail to meet their initial criticisms, affirming “decisions to change or replace a classification system should be based on the results of scientific competition (e.g., tests of incremental) validity” (2022, 288), which they judge HiTOP to be incapable of producing.

Setting aside that neither group provides an account of validity, this version of a back and forth between frameworks is compromised since all parties are operating on entirely different standards as well as a fundamental conception for what counts as validity. As a result, we cannot presently adjudicate between approaches and conclude as Haeffel et al. (Reference Haeffel, Jeronimus, Fisher, Kaiser, Lesley Jo Weaver, Goodson, Soyster and Lu2022) do that one approach is less valid than the other. Similar to how Zachar, Krueger, and Kendler describe the disparate approaches to the classification of personality disorders in DSM-5, they are “like two ships passing in the night” (Reference Zachar, Krueger and Kendler2016, 5).

5.2. Difficulties coordinating and integrating between frameworks

A second implication is the prevention of coordinating and integrating between differing approaches. Despite each approach maintaining that their own framework is the best, there have been several proposals to suggest how certain approaches may be successfully linked. For example, Michelini et al.’s (Reference Michelini, Palumbo, DeYoung, Latzman and Kotov2021) “Linking RDoC and HiTOP: A New Interface for Advancing Psychiatric Nosology and Neuroscience” suggests RDoC, with its biologically based focus, may provide helpful tools for “elucidating the underpinnings of the clinical problems in HiTOP” (2021, 4), whereas HiTOP may motivate RDoC studies “by providing psychometrically robust clinical targets” (2021, 4).

Any interface between HiTOP and RDoC assumes the potential for achieving construct stabilization. Construct stabilization, defined by Sullivan (Reference Sullivan2016a, Reference Sullivan2016b), is the coordination across researchers situated in the same and different approaches to come to agreements regarding (i) how to generally define terms designating constructs, (ii) the best experimental paradigms for studying a given construct, and (iii) the conditions under which two experimental paradigms can be said to measure the same construct. Coordination and integration between frameworks are thought to be contingent on the ability to stabilize constructs.

The problem is, given distinct conceptions of validity, conditions (ii) and (iii) of construct stabilization cannot be met. Condition (ii) cannot be met because both HiTOP (structure-first) and RDoC (biology-first) maintain that their own approach informs the best experimental paradigms and, given these approaches are so disparate, they ultimately will not coalesce on specific paradigms. Condition (iii) cannot be met since differing standards for what is required for validation dictate diverging standards as to acceptable ways to measure HiTOP or RDoC constructs.

Perhaps the best argument against coordination and integration is that RDoC and HiTOP constructs do not even refer to the same thing. HiTOP constructs are understood as pure constructs, i.e., data-driven factors based on statistical output from factor analysis of psychological testing data. For RDoC, constructs are empirical functions selected via a process of expert curation and are thought to refer to specific biological and cognitive processes (e.g., Reward Learning). Thus, even if researchers were to come together to discuss what they mean by “construct,” without dramatically overhauling their entire approach, construct stabilization across frameworks, and thus coordination or integration, isn’t currently achievable.

5.3. Difficulties in establishing a unified concept of validity

A third implication is that the approaches will be unable to establish a unified concept of validity. Take the broad concept of depression for which researchers wish to develop valid psychiatric classifications. Ultimately, the presence of disparate conceptions of depression amounts to a troubling scenario for trying to do any kind of unified validation. The central issue is that the approaches don’t agree on the phenomenon they’re trying to make inferences about. If they don’t agree on what the features or attributes are, progress in achieving a unified conception of validity for psychiatric classifications is incredibly difficult due to its preceding influence on each approach’s standards of validation.

A similar problem arises in psychological testing, whereby if multiple testers are attempting to design their own inventory to measure “intelligence,” if there is a significant enough difference in the understanding of what “intelligence” amounts to, then developing and comparing valid tests of “intelligence” becomes incredibly difficult. Those who view intelligence in one way will not just want but need to pursue one conception of validity with specific standards. In contrast, another approach, finding the original approach misguided will reject it completely and insist on pursuing validation in a different manner.

Thus, starting with disparate conceptions of psychiatric disorders as is the case across DSM, HiTOP, the Network Approach, and RDoC influences the development of divergent standards for validating psychiatric classifications. How each approach conceptualizes and represents the phenomenon necessitates how its researchers will study, evaluate, and make inferences concerning validating evidence, ultimately shaping any conception of validity achieved.