1) Not checking assumptions

SEM relies on a range of assumptions that operate at different levels, including assumptions about model specification, estimation, data characteristics, and study design. Some of these assumptions are specific to the estimator used, whereas others are shared with more general statistical models. Because the relevance of particular assumptions depends on the type of data, the model structure, and the analytical choices made by the researcher, not all assumptions apply equally in every SEM study. In this section, we highlight some assumptions that commonly arise in applied SEM research and that have direct implications for model estimation, missing data handling, and sample size. Readers who are interested in more detailed and technical discussions of SEM assumptions are referred to standard methodological treatments (e.g., Hancock & Schoonen, Reference Hancock and Schoonen2015; Kline, Reference Kline2023; West et al., Reference West, Wu, McNeish, Savord and Hoyle2023).

Classical SEM estimation and model-fit testing were developed under asymptotic assumptions, particularly assuming a large number of observations relative to the number of variables in the model (e.g., Marcoulides et al., Reference Marcoulides, Yuan, Deng and Hoyle2023). Under such conditions, key statistical assumptions are more likely to be met, and commonly used test statistics behave as intended. In applied research, however, these ideal conditions are often not satisfied. Researchers frequently work with sample sizes smaller than those assumed by asymptotic theory, as well as with complex models containing many observed variables, non-normal data, and missing responses. When this mismatch between assumptions and data is not acknowledged or addressed, it can lead to estimation problems and misleading conclusions, even when the model appears to fit the data well.

In light of these considerations, one of the first assumptions to consider concerns the distribution of the observed variables and the choice of estimator. When using normal-theory maximum likelihood (ML) estimation, an important assumption is that the observed continuous variables follow a multivariate normal distribution. Data normality can be examined using univariate and multivariate indicators such as skewness and kurtosis. When this assumption is violated for continuous outcomes, robust corrections such as the Satorra–Bentler scaled chi-square and robust standard errors provide a more appropriate basis for inference (e.g., Marcoulides et al., Reference Marcoulides, Yuan, Deng and Hoyle2023). For models involving categorical observed variables, estimation approaches based on diagonally weighted least squares, such as the weighted least squares mean and variance adjusted (WLSMV) procedure, are generally preferred (e.g., Kline, Reference Kline2023).

An important implication of estimator choice is that both the relevant distributional assumptions and the available options for handling missing data differ across modeling frameworks. For models estimated using WLSMV, normality is not assumed for the observed categorical variables themselves, but rather for the latent response variables that are assumed to underlie these observed responses (Li, Reference Li2015). Latent variable normality cannot be directly assessed using empirical diagnostics in the same way as observed-variable normality; instead, its implications must be evaluated indirectly through model fit, robustness checks, and transparent reporting of modeling assumptions. Estimator choice also constrains how missing data can be handled. Full information maximum likelihood (FIML) is available when using normal-theory ML estimation and provides an effective approach for handling data assumed to be missing completely at random (MCAR) or missing at random (MAR). When models involve categorical observed variables and are estimated using WLSMV, FIML is typically not available; in such cases, alternative approaches such as pairwise maximum likelihood can be used to address missing data under MCAR or MAR assumptions (Katsikatsou et al., Reference Katsikatsou, Moustaki and Jamil2022).

Beyond estimator-specific assumptions, SEM also relies on a number of general statistical assumptions that it shares with other modeling approaches based on the general linear model. One key assumption is the independence of observations, meaning that each participant’s responses are assumed to be unrelated to those of other participants. This assumption is often violated in applied research, for example, when data are collected from clustered settings such as students nested within classes, schools, or institutions, or when convenience or snowball sampling is used. When such dependencies are ignored, standard errors and test statistics may be biased, leading to incorrect inferences. Another common concern is sensitivity to outliers, which can disproportionately influence parameter estimates and model fit. In addition, SEM assumes homoskedasticity, or equal error variances across observations or groups. When heteroskedasticity is present, standard errors may be inaccurate, even if the overall model appears to fit the data well. In cases where these assumptions are violated, alternative approaches such as multilevel SEM or the use of cluster-robust standard errors and test statistics can be used to address these issues.

Another consideration that directly affects estimation quality and model evaluation concerns sample size. Many SEM estimation methods rely on large-sample properties, and when sample sizes are small relative to model complexity, parameter estimates, standard errors, and fit statistics may become unstable (Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018; Feng & Hancock, Reference Feng, Hancock and Hoyle2023; Morrison et al., Reference Morrison, Morrison and McCutcheon2017). In applied research, practical constraints such as limited access to participants, time, or funding often restrict achievable sample sizes. As a result, researchers have relied on simple heuristics, such as minimum sample sizes of 100 or 200 (Boomsma, Reference Boomsma1985) or the rule of ten observations per variable (Everitt, Reference Everitt1975). However, such rules of thumb provide only rough guidance, as the required sample size depends on multiple interacting factors. Simulation research has shown that sample size requirements vary as a function of factor loadings, the number of indicators per construct, and the presence of missing data. For example, Wolf et al.’s (Reference Wolf, Harrington, Clark and Miller2013) simulation showed that in a one-factor model with factor loadings of .50, a minimum sample size of approximately 200 was required when the factor had 3 or 4 indicators, whereas a sample of 100 was sufficient when the factor had 6 or 8 indicators. When factor loadings were .80, the required minimum sample size dropped to approximately 50 across all tested conditions. Moreover, the presence of missing data further increased the required minimum sample size, ranging from 200 to 300, depending on the extent of missingness. These findings highlight that sample size should be considered in relation to the specific model and data characteristics rather than evaluated against fixed numerical thresholds.

Given this complexity, sample size decisions are better treated as part of the overall model planning rather than evaluated using fixed cutoff values. One practical approach is to use simulation-based power analysis, which allows researchers to examine how a specific model is expected to perform under conditions that are similar to the intended study (Feng & Hancock, Reference Feng, Hancock and Hoyle2023). Through simulation, it is possible to evaluate power for overall model fit as well as for individual parameters, while taking into account features such as model structure, effect sizes, and missing data. The simsem package in R (Pornprasertmanit et al., Reference Pornprasertmanit, Miller, Schoemann, Jorgensen and Quick2021) provides one way to conduct such analyses across different SEM conditions. Using this approach, researchers can make more informed decisions about sample size and better evaluate whether the assumptions underlying their analyses are likely to hold (see also online supplementary materials).

2) Not validating a measurement model

The use of psychometrically sound measures is vital (e.g., Kline, Reference Kline2023; Ockey & Choi, Reference Ockey and Choi2015). In the context of L2 research, Plonsky and Derrick (Reference Plonsky and Derrick2016) examined the percentage of reliability estimates reported across various L2 subdomains, revealing that reliability was documented in 6% to 64% of studies (see also Kostromitina et al., Reference Kostromitina, Sudina and Baghlaf2025; Sudina, Reference Sudina2021, Reference Sudina2023). This suggests that reliability reporting varies across subdomains. When two measures exhibit a low correlation, it is unclear whether the constructs themselves are weakly related or whether poor measurement reliability is responsible for the weak association.

In SEM, analytical models are generally composed of measurement models and structural models. Measurement models specify the relationship among observed variables and latent (unobserved) variables, thereby assessing the extent to which constructs are effectively operationalized and measured through their indicators. Structural models, in contrast, focus on the relationships among latent constructs. Given that researchers are often primarily interested in how constructs are related, structural models are often a focal point in SEM analyses.

For structural models to be interpreted with confidence, however, the measurement model must first demonstrate good fit to the data. Poor measurement model fit suggests that the constructs may not be measured appropriately, which in turn calls into question the validity of interpreting relationships among constructs in structural models. This is analogous to ensuring that instruments such as tests and questionnaires are valid and reliable before interpreting their scores or relationships among them.

One practice that can complicate model evaluation involves modeling the measurement and structural models simultaneously rather than sequentially (the latter of which involves testing the measurement models independently before testing the structural models). This practice leads to less transparency because, if a simultaneous model shows poor fit, it is unclear whether the issue originates from the measurement models, the structural models, or both. In contrast, when researchers first ensure that the measurement models fit well, they can more confidently attribute any remaining issues to the structural models, thereby refining their theoretical understanding.

Questions about measurement model quality also extend to how reliability is assessed within SEM. Computing Cronbach’s alpha is not a requirement when using SEM. In fact, it is often better to use alternative approaches to reliability that are more suitable for SEM. Some reliability measures, such as Cronbach’s alpha coefficient, rely on the assumption of equal item precision (tau-equivalence), which in factor-analytic terms corresponds to equal factor loadings, an assumption that is often violated in practice (Raykov, Reference Raykov2004). SEM-based approaches, particularly those from confirmatory factor analysis (CFA), offer more accurate alternatives. One such approach is Composite Reliability (CR), which is a more accurate estimate of internal consistency because it accounts for the actual factor loadings of each item (Fornell & Larcker, Reference Fornell and Larcker1981). For further details on Cronbach’s alpha and CR, see Peterson and Kim (Reference Peterson and Kim2013) and Raykov (Reference Raykov1998). Another approach is Average Variance Extracted (AVE), which assesses the amount of variance captured by the construct relative to measurement error (Fornell & Larcker, Reference Fornell and Larcker1981). Using CR and AVE enables researchers to better assess a scale’s reliability and convergent validity within the specific context of their SEM model, and AVE is also commonly used as part of procedures for evaluating discriminant validity.

3) Not testing competing models

In many cases, it is possible to think of alternative models that might explain the data equally well, if not better. Not testing these models gives the reader no choice but to accept the model hypothesized by the authors. This is problematic because the author’s model is usually only one of the different models potentially supported by the data. Comparing multiple models allows researchers to critically assess the theoretical framework and its application within the model (e.g., Yun et al., Reference Yun, Hiver and Al-Hoorie2018). Metrics such as Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) can help compare competing models to determine which one provides the best fit to the data while maintaining parsimony. Selecting the most appropriate model after empirically and theoretically eliminating competing alternatives is more meaningful than testing and selecting a single model in isolation (Joe et al., Reference Joe, Hiver and Al-Hoorie2017).

Testing competing models allows researchers to evaluate both the strength of the underlying theoretical rationale and its statistical implementation. For instance, in their review of the L2 Motivational Self System, Hiver and Al-Hoorie (Reference Hiver and Al-Hoorie2020) argued that the three components of the model—the ideal L2 self, the ought-to L2 self, and the L2 learning experience—may be modeled in various ways. Figure 1 presents four alternative models. They can be modeled as (a) a correlated three-factor model, (b) a correlated three-factor model with cross-loadings, (c) a bifactor model without cross-loadings, or (d) a bifactor model with cross-loadings. In the latter two cases, a general factor exerts influence on all items in the dataset. In some cases, cross-loadings may be theoretically expected because, for example, motivational self-constructs are conceptually related, and items may reflect overlapping aspects of learners’ self-guides. Detailed treatments of competing model evaluation within SEM, including alternative measurement specifications, are found in Morin (Reference Morin and Hoyle2023) and Morin et al. (Reference Morin, Myers, Lee, Tenenbaum and Eklund2020).

Figure 1. Four competing models of L2 Motivational Self System. Id = ideal L2 self, Ou = ought-to L2 self, ATLE = attitude toward L2 learning experience, Mtv = Motivation.

Equally important to consider is how to compare models. One key issue is the evaluation of measurement equivalence/invariance. Measurement invariance testing—conducted by comparing nested models (a less constrained vs. a more constrained model) and assessing changes in fit indices such as root mean square error of approximation (RMSEA) and comparative fit index (CFI)—is a widespread application in SEM and an important step in ensuring that constructs are measured equivalently across different conditions (e.g., Kline, Reference Kline2023). Such conditions may include groups (e.g., L1 vs. L2 learners), methods (e.g., questionnaire vs. interview formats), and time points (e.g., Time 1 vs. Time 2). Failing to conduct or report invariance testing can lead to misleading conclusions. Savalei et al. (Reference Savalei, Brace and Fouladi2024) review ongoing debates on this issue and recommend the use of RMSEA values associated with the chi-square difference test, rather than relying solely on the chi-square difference test or on changes in RMSEA.

4) Making insufficiently justified modeling decisions

SEM provides researchers with a flexible framework to model relationships among variables. While SEM has been categorized into exploratoryFootnote ¹ and confirmatory approaches in the literature, most applications fall somewhere in between. It is also possible to use both approaches with the same data set (Morin, Reference Morin and Hoyle2023). The exploratory approach relies on data-driven decisions to model relationships between variables without a strong theoretical basis. For example, a researcher may specify that a latent variable influences all observed variables. They may also specify directional, causal relationships among latent variables without prior theoretical justification. In contrast, the confirmatory approach is guided by theoretical foundations. This approach may involve specifying that a latent variable influences only certain observed variables rather than all of them. Similarly, it may define directional relationships based on prior theory, ensuring one latent variable influences another but not vice versa. If the model-building process is not clearly reported, readers will be left uncertain about how constructs have been operationalized in SEM and how trustworthy the results are. Nevertheless, although SEM can be used in both an exploratory and confirmatory manner, we stress prioritizing the confirmatory approach when possible. One reason is that SEM allows users to take a theory-based, causal approach. This is particularly evident when an arrow is drawn from one variable to another, indicating a directional, cause–effect relationship that should be grounded in theory or prior research (see Al-Hoorie & Hiver, Reference Al-Hoorie and Hiver2025, for more detail on causal inference from observational data).

It is worth noting that exploratory research, when conducted ethically, does not in itself constitute a QRP. The critical issue lies in ensuring that post hoc modeling decisions remain theory-guided and data-supported, rather than merely data-driven, while simultaneously considering how empirical evidence can inform the refinement of theory. As discussed above, research rarely falls neatly into purely exploratory or purely confirmatory categories; instead, it spans a continuum between these extremes. The most reliable scientific inferences arise not when data dictates questions or hypotheses after the fact, but when theoretically grounded questions are posed in advance and then empirically tested in order to minimize the risk of post hoc significance chasing. Any data-informed decisions about refining a theoretical model, even when conducted rigorously, require newly formulated hypotheses to be tested on new data, as evaluating such hypotheses using the same data that generated them does not constitute an independent test. This process, commonly referred to as cross-validation, is essential for drawing robust scientific inferences. One way to further increase transparency is preregistration, which allows researchers to make key modeling decisions explicit before data analysis.

As an illustration, in their analysis of studies on the L2 Motivational Self System, Hiver and Al-Hoorie (Reference Hiver and Al-Hoorie2020) reviewed how different studies approached causal modeling in SEM. For example, Taguchi et al. (Reference Taguchi, Magid, Papi, Dörnyei and Ushioda2009) examined the influence of (a) ideal L2 self on (b) L2 learning experience (which they called “attitudes toward learning English”), whereas Kormos and Csizér (Reference Kormos, Csizér, Dörnyei and Ushioda2009) investigated the impact of (b) L2 learning experience on (a) ideal L2 self. Despite using the same constructs, these studies specified opposite causal directions. Given that both models demonstrated data fit, this may suggest the possibility of bidirectional causality: a → b and b → a. Alternatively, as Hiver and Al-Hoorie (Reference Hiver and Al-Hoorie2020) argued, the findings may also reflect reciprocal causality, wherein both variables exert mutual influence (e.g., Kline, Reference Kline2023): a ⇄ b. It is also possible that no causal relationship exists, and the observed association is simply caused by a third variable (see Al-Hoorie & Hiver, Reference Al-Hoorie and Hiver2025).

Among the three types of causality, the first two cases, a → b and b → a, produce the same statistical results under specific conditions, because the models are statistically equivalent in terms of the model-implied covariance matrix and mean vector (see online supplementary materials). This equivalence reinforces the importance of robust theory over statistical results alone. For example, Hiver and Al-Hoorie (Reference Hiver and Al-Hoorie2020) demonstrated that two models describing the relationship between (a) ideal L2 self and (b) disappointment in daily life produced identical results (i.e., path coefficients and fit indices; see Figure 2), even though the direction of the path between these two variables was reversed. In one model, the ideal L2 self appeared to be influenced by disappointment, whereas in the other model, the ideal L2 self itself seemed to influence disappointment. This result arises because only the directional path between the predictor and mediator was changed, while the remainder of the model remained unchanged. In contrast, reversing the direction of the path between a mediator and an outcome would affect parameter estimates, even though the models are still statistically equivalent and produce identical model-implied covariance matrices and mean vectors. Thus, obtaining an acceptable model fit does not support a unique, causal interpretation, stressing the need for theoretical justification when specifying paths.

Figure 2. Two models with different causal specifications resulting in identical structural coefficients and identical model fit. Error terms have been removed for simplicity.

Stronger causal inference in SEM depends not only on statistical modeling but also on theory and research design, including careful consideration of confounders, mediators, and colliders (Al-Hoorie & Hiver, Reference Al-Hoorie and Hiver2025). Longitudinal designs are particularly useful because they impose temporal ordering and allow researchers to examine change over time. Models such as the random-intercept cross-lagged panel model, the stable trait, autoregressive trait, and state model, and related dynamic panel approaches provide tools for evaluating directional hypotheses using repeated measures (Lucas, Reference Lucas2023). However, even with longitudinal data, causal interpretations cannot be based on model fit alone and must remain grounded in theory and explicit modeling assumptions.

5) Modifying the model post hoc

Unlike insufficiently justified modeling decisions discussed earlier, where the researcher specifies the model without theoretical justification, here the researcher modifies or respecifies the model based on the results. This is often achieved using modification indices to enhance model fit. For instance, modification indices may indicate the need to add, remove, or modify paths between variables, correlate residual errors, or incorporate cross-loadings. While these changes can enhance model fit over several iterations, implementing them without a strong theoretical basis compromises the validity of the model and may constitute a QRP. Rather, such modifications should serve as signals that encourage the researcher to reconsider the theoretical model.

A key distinction must be drawn between data-driven decisions and data-supported theoretical updates. In data-driven model modification, researchers revise the model primarily in response to statistical patterns (e.g., adding paths with large modification indices), which can encourage ad-hoc rationalizations and weaken the confirmatory interpretation of SEM results. In contrast, theory-based refinement treats statistical indications as prompts to reconsider the theoretical model. For example, patterns in correlation residuals may motivate a theoretically grounded explanation for why certain variables are more or less strongly related than expected. Only when such an explanation is plausible should a corresponding modification be considered. Even when modifications are introduced post hoc, this reasoning can be made more rigorous by asking whether the same path or covariance would have been specified if the model were formulated a priori. In some cases, alternative or nested models can also be specified in advance, allowing later model comparisons to be treated as confirmatory rather than purely exploratory.

Removal of parameters or variables solely on the basis of statistical non-significance or small sizes is similarly not justifiable. If these elements are theoretically relevant to the construct, as hypothesized a priori, they should be retained in order to align with the confirmatory nature of the model. Their exclusion may result in an underrepresentation of the construct, thereby diminishing the model’s interpretability and theoretical validity.

Further, making post hoc modifications to models can result in overfitting, where the model fits the current dataset well but may not generalize to new data. This may raise concerns in cross-validation and the broader applicability of the substantive findings (e.g., West et al., Reference West, Wu, McNeish, Savord and Hoyle2023). Given that the modified model deviates from the original, theory-driven model, it should ideally be tested with an independent dataset to determine the extent to which it remains supported (e.g., Kline, Reference Kline2023; Leeming et al., Reference Leeming, Vitta, Hiver, Hicks, McLean and Nicklin2024).

Therefore, post hoc modifications should be theoretically justified. Because statistical software provides these suggestions based on statistical rather than theoretical considerations, they should be carefully reviewed and applied only when they hold substantive meaning. Otherwise, the model may achieve a strong statistical fit but lack substantive value. While we emphasize the risks associated with post hoc modifications, we also acknowledge that data often challenge theoretical expectations and that exploratory insights can play a constructive role in theory development. It is important to recognize, however, that post hoc model modifications generate new hypotheses using information from the data. As mentioned above, these hypotheses require testing on new data, as evaluating them on the same dataset does not constitute a valid confirmatory test. Adopting a transparent and self-skeptical approach helps maintain scientific integrity while still allowing theory to be refined and extended.

6) Overemphasizing global model fit indices

Model fit in SEM is assessed at both global and local levels. Overemphasizing global fit refers to evaluating model adequacy primarily or exclusively on the basis of global fit indices, while neglecting residuals, localized misfit, and substantive theory. Global fit evaluates the overall alignment of the model with the data, relying on indices such as the chi-square test statistic (and its p-value for given degrees of freedom) and several indices of approximate fit: RMSEA, CFI, and standardized root mean square residual (SRMR). Local fit, by contrast, assesses the degree to which the model adequately reproduces relationships between each pair of variables. While global fit provides a broad measure of model-data consistency, it does not diagnose the specific areas where misfit may occur. Local fit addresses this limitation by examining the individual associations within the model, allowing researchers to pinpoint and help address discrepancies. It is typically evaluated through standardized residuals or residual covariance matrices, which help identify specific variable pairs or parameters that contribute disproportionately to overall misfit. Given that global and local fit statistics assess models from different perspectives, both should be incorporated into model evaluation (e.g., Appelbaum et al., Reference Appelbaum, Cooper, Kline, Mayo-Wilson, Nezu and Rao2018; Kline, Reference Kline2023, Reference Kline2024). While global fit indices are more commonly reported, disregarding local fit may result in models that appear well-fitting at a global level but contain numerous localized inconsistencies. Models with statistically significant global misfit might be rejected without considering local fit indices, which could indicate a major discrepancy (or several of them), the accumulation of many minor (potentially ignorable) discrepancies, or both, thereby providing a more nuanced understanding of data–model fit.

Kline (Reference Kline2023) recommends an approach to model evaluation that incorporates both global and local fit. If the chi-square test is statistically significant, indicating that the model may not fit the data well, tentatively reject the model and examine the residuals (i.e., local fit) to identify discrepancies between the model and the data. If no substantial discrepancies are detected, evaluate the model substantively and empirically. For empirical evaluation, consider RMSEA, CFI, and SRMR. If the chi-square is statistically non-significant, suggesting that the model may fit the data, inspect the residuals. If no major inconsistencies are found, evaluate the model substantively and empirically using these fit indices.

Kline’s (Reference Kline2023) approach to model evaluation merits further discussion in two points. First, regardless of whether the chi-square is statistically significant, it is critical to examine residuals. Residual analysis identifies differences between the model and the data at the level of individual variable pairs, allowing researchers to locate misfit and modify the model accordingly. Since global fit indices are not designed to reveal these discrepancies, a model should be rejected even if the chi-square is non-significant if large residuals are present (for more on residual analysis, see Kline, Reference Kline2024). Second, for models with statistically significant chi-squares, Kline (Reference Kline2023) proposed to tentatively reject the model and inspect the residuals. This approach places less emphasis on statistical significance and instead encourages researchers to diagnose whether misfit is due to large residuals in specific parts of the model.

It is important to clarify the interpretation of chi-square significance. The chi-square test does not detect sample size; rather, when the null hypothesis of exact fit is false, the test becomes increasingly capable of detecting even very small discrepancies as sample size increases (e.g., West et al., Reference West, Wu, McNeish, Savord and Hoyle2023). This behavior is not unique to SEM: for example, an independent-samples t-test gains power to detect increasingly small differences as sample size increases. In other contexts, researchers accompany statistical tests with effect-size measures to distinguish statistically significant but practically negligible results. In SEM, approximate fit indices were originally intended to serve this function by providing information about the magnitude of model misfit, rather than replacing statistical tests or local fit diagnostics. A common misunderstanding—that a significant chi-square “reflects sample size more than misfit”—may have led researchers to dismiss chi-square significance entirely. Recognizing that chi-square significance provides meaningful information about model misfit, while approximate fit indices and residuals provide complementary perspectives on its magnitude and location, helps avoid this misunderstanding and supports more defensible model evaluation.

Relatedly, evaluating models rigidly based on fit index cutoff criteria is not a recommended approach. This is because the studies that established these thresholds may not be directly applicable to all research contexts. For instance, Hu and Bentler’s (Reference Hu and Bentler1999) recommended cutoffs—CFI close to .95, RMSEA close to .06, and SRMR close to .08—are frequently used to evaluate models somewhat uncritically. However, these cutoffs were derived from a simulation study using confirmatory factor models with 3 factors and 15 observed variables, tested under specific conditions. Unless a researcher’s model and study conditions are similar to those in Hu and Bentler’s (Reference Hu and Bentler1999) study, these cutoffs should not be applied uncritically. In fact, Hu and Bentler (Reference Hu and Bentler1999) viewed these values as general guidelines rather than absolute rules. Furthermore, the cutoffs were not supported in later studies, questioning their applicability across different modeling conditions (e.g., Fan & Sivo, Reference Fan and Sivo2005; Yuan, Reference Yuan2005).

To address these challenges, one approach is to examine dynamic, study-specific cutoffs rather than strictly adhering to Hu and Bentler’s (Reference Hu and Bentler1999) criteria. Simulation-based approaches can be used not only for power and sample size planning, but also to evaluate how fit indices are expected to behave under a given model, sample size, and degree of misspecification. This strategy does not diminish the importance of local fit inspection or theoretical justification in model evaluation (see McNeish & Wolf, Reference McNeish and Wolf2023, for further discussion; see also online supplementary materials).

7) Incomplete or nontransparent reporting

As discussed above, SEM provides a flexible framework for modeling relationships among variables. This flexibility extends to the wide range of output options available, which depend on the type of model analyzed and the additional details requested. For example, models with a large number of variables produce extensive lists of parameter estimates. If the dataset contains non-normal data, the output may include results both with and without adjustments for nonnormality. While these analytical capabilities allow for more comprehensive analyses, they also introduce challenges in presenting results clearly. This complexity increases the risk of selective reporting, where findings that align with the researcher’s hypothesis are prioritized while contradictory results are omitted. Selective reporting, as well as incomplete or nontransparent reporting of the issues discussed above, compromises the transparency and rigor of model evaluation and interpretation.

Researchers should therefore strive for complete and transparent reporting. Rather than relying solely on data-driven decisions, they should provide theoretical justifications for model specification. Modifying the model based on results is appropriate only if the changes are substantively supported and transparently documented. Testing measurement models is an important step in model development and should be systematically reported. Evaluating competing models allows researchers to test hypotheses, and documenting this process contributes to theoretical refinement. Reporting both global and local fit statistics indices provides a comprehensive assessment of model–data fit. Because fixed cutoffs for fit indices may not always be appropriate in a given study context, incorporating dynamic cutoffs can offer additional insights. Lastly, reporting whether the necessary assumptions for SEM were met—and describing how any violations were addressed—enhances the credibility of the research.

Ensuring complete and transparent reporting is a fundamental principle of research (Al-Hoorie et al., Reference Al-Hoorie, Cinaglia, Hiver, Huensch, Isbell, Leung and Sudina2024; Liu et al., Reference Liu, Chong, Marsden, McManus, Morgan-Short, Al-Hoorie, Plonsky, Bolibaugh, Hiver, Winke, Huensch and Hui2023). However, practical constraints, such as word limits, may restrict the amount of information that can be included in published studies. Even when supplementary files are provided or data are made available through repositories such as IRIS (https://www.iris-database.org) and Open Science Framework (OSF; https://osf.io/), essential details may still be missing, or additional information may be required for future analyses. Moreover, some readers may seek to reproduce the reported models. To enhance reproducibility, researchers are encouraged to share analysis code, data, and output files whenever feasible (though see Chapelle & Ockey, Reference Chapelle and Ockey2024, for potential drawbacks associated with this practice).

Ethical concerns may prevent researchers from sharing raw data. However, they can still provide summary statistics, such as means, standard deviations, and correlations, which allow other researchers to reproduce the reported model. SEM software calculates these values during analysis, meaning that summary statistics can sometimes be sufficient for reproducing models. Thus, reproducing SEM results without raw data is possible under certain conditions, such as when the dataset is normally distributed and complete (i.e., contains no missing responses). If these conditions are not met, raw data are necessary to accurately reproduce the original model. To determine whether a model can be reproduced using summary statistics alone, researchers can compare the consistency of results derived from both raw data and summary statistics-based analyses. Regardless of whether raw data can be shared, reporting summary statistics is strongly recommended, as it provides readers with a clear overview of the dataset’s characteristics.

A recent example is provided by Nagle (Reference Nagle2023), who shared data, analysis code, and output files. In examining longitudinal change in individual difference variables, the study addressed the issue of longitudinal measurement invariance. To illustrate this approach in motivational research, the author tested invariance for the ought-to L2 self using data from 141 U.S. university students collected at three time points across a semester (beginning, midterm, and end). Results supported strong invariance, indicating the same factor structure, equal loadings, and equal intercepts across time. These findings suggest that the ought-to L2 self was measured consistently, and that observed changes reflected true change in the construct rather than changes in measurement. We ran the R code and successfully reproduced the results reported in the study.

Another example is Suzuki and Kormos (Reference Suzuki and Kormos2023), who examined the relationship between cognitive fluency and utterance fluency in second language oral proficiency among 128 Japanese learners of English. After examining measurement models for both constructs, they tested a structural model linking them. One key finding was that speed fluency—one component of utterance fluency—was related to linguistic resources—one component of cognitive fluency—only in certain task types (e.g., reading-to-speaking), highlighting the moderating role of task characteristics. Using the shared materials, we successfully reproduced the reported analyses and results.

Table 1 provides an overview of the main QRPs discussed in this article, along with corresponding recommendations.

Table 1. SEM QRPs and Best Practice Recommendations