The use of the observational methodology in study designs allows the spontaneous behaviors of humans in natural contexts to be recorded and quantified (Anguera et al., Reference Anguera, Blanco-Villaseñor, Losada and Sánchez-Algarra2020). Such methods are frequently employed in psychological research as they require minimal intervention and are independent from standardized measurement tools, thus enabling the collection of unbiased information (Anguera et al., Reference Anguera, Blanco-Villaseñor, Losada and Portell2018).
The observational methodology described above differs from that used in the observational studies typically found in health research. These studies are classified by quantitative (QUANT) design type, including cohort, case–control, and cross-sectional studies. In these studies, empirical groups are compared to identify cause–effect relationships when the feasibility of randomization and experimental control is limited (Cochran & Chambers, Reference Cochran and Chambers1965). In contrast, observational methodology is considered a mixed-methods approach (Anguera et al., Reference Anguera, Camerino, Castañer, Sánchez-Algarra and Onwuegbuzie2017), given that it transforms and connects both qualitative (QUAL) and QUANT elements (Cresswell & Plano-Clark, Reference Cresswell and Plano-Clark2017). Specifically, research designs based on observational methodology can be organized in a QUAL–QUANT–QUAL sequence. The initial QUAL phase involves the study design, the choice of the observation instrument, and the systematic recording of QUAL data. The subsequent QUANT phase focuses on transforming the QUAL data into a code matrix and conducting data quality control procedures prior to data analysis. Finally, the last QUAL phase involves interpreting the results within the context of the initial problem and previous literature (Anguera et al., Reference Anguera, Blanco-Villaseñor, Losada and Sánchez-Algarra2020).
Over the past decade, the increase in studies based on observational methodology has prompted numerous researchers to direct their attention to the quality concerns associated with this mixed-methods approach (Chacón-Moscoso et al., Reference Chacón-Moscoso, Sanduvete-Chaves, Portell and Anguera2013, Reference Chacón-Moscoso, Anguera, Sanduvete-Chaves, Losada and Portell2019; Portell et al., Reference Portell, Anguera, Chacón-Moscoso and Sanduvete-Chaves2015). This problem could be addressed by ensuring homogeneity and transparency in both studies that rely on observational methodology and in the criteria established for assessing their quality (Chacón-Moscoso et al., Reference Chacón-Moscoso, Sanduvete-Chaves, Portell and Anguera2013).
In order to achieve this, the Methodological Quality Scale for Studies Based on Observational Methodology (MQSOM) was developed (Chacón-Moscoso et al., Reference Chacón-Moscoso, Anguera, Sanduvete-Chaves, Losada and Portell2019; Sanduvete-Chaves et al., Reference Sanduvete-Chaves, López-Arenas, Anguera and Chacón-Moscoso2025), drawing on the Guidelines for Reporting Evaluations Based on Observational Methodology (GREOM) (Portell et al., Reference Portell, Anguera, Chacón-Moscoso and Sanduvete-Chaves2015). The MQSOM is comprised of 11 items that assess a second-order dimension of global methodological quality through two first-order dimensions: the quality of design and the quality of measurement and analysis. It specifies the minimum methodological aspects necessary to improve the quality of studies based on observational methodology. Adequate reliability and validity evidence based on its internal structure was obtained.
The main objective of this study was to obtain empirical evidence of the convergent and discriminant validity of the MQSOM when compared with other instruments that measure methodological quality in mixed-methods studies.
Method
Participants (Units of Analysis)
The 96 papers examined for the validation of the MQSOM were obtained from among 650 studies found in an exhaustive search completed for a previous investigation (Sanduvete-Chaves et al., Reference Sanduvete-Chaves, López-Arenas, Anguera and Chacón-Moscoso2025). All met the following inclusion criteria: (a) The observational methodology was used, (b) the work was original, (c) the publication presented the usual sections of a research article (introduction, method, results, and discussion), and (d) it was written in English or Spanish. Consequently, this study employed stratified random sampling to guarantee a homogeneous distribution of the units of analysis across the range of MQSOM scores. The required sample size was calculated using G*Power for 90% power and a 5% type I error rate to detect a standardized effect size of 0.4 or larger, resulting in a sample size of 96 papers.Footnote 1
Instruments
The MQSOM (Sanduvete-Chaves et al., Reference Sanduvete-Chaves, López-Arenas, Anguera and Chacón-Moscoso2025) was used for this study. The scale has adequate psychometric properties (root-mean-square error of approximation [RMSEA] = .000; non-normed fit index [NNFI] = 1; goodness of fit index [GFI] = .98; adjusted goodness-of-fit index [AGFI] = .97) and comprises a second-order methodological quality dimension (ω = .87; D = .55) that contains two first-order dimensions: quality of design (6 items; ω = .90; D = .46; intraclass correlation coefficient [ICC] = .933–.967) and quality of measurement and analysis (5 items; ω = .68; D = .67; ICC = .797–.988). Each item receives a score for methodological quality levels from 0 (lowest methodological quality) to 1 (highest methodological quality).
For the comparison with the MQSOM, three instruments were selected: (a) the Methodological Rigor in Mixed Methods (MRMM) checklist (Harrison et al., Reference Harrison, Reilly and Creswell2020), the GREOM (Portell et al., Reference Portell, Anguera, Chacón-Moscoso and Sanduvete-Chaves2015), and the Mixed Methods Appraisal Tool (MMAT) checklist (Hong et al., Reference Hong, Fàbregues, Bartlett, Boardman, Cargo, Dagenais, Gagnon, Griffiths, Nicolau, O’Cathain, Rousseau and Vedel2018). Table 1 presents an overview of the instruments chosen for the comparison.
Overview of the instruments for comparison
Note. FWCI = field-weighted citation impact.
Procedure
A total of nine instruments were considered for comparison to the MQSOM. The selected instruments are either guidelines or checklists from both the social and health sciences. The selection criteria were (a) type of studies the instruments evaluate (tools that assessed studies based on observational methodology and mixed-method studies were considered); (b) whether the instrument addresses the quality of reporting, design, measurement, and analysis; and (c) reliability and/or validity evidence reported. Table 2 presents the characteristics of the instruments considered.
Characteristics of the instruments considered for comparison to the MQSOM
Note. EGPQRS = Evolving Guidelines for Publication of Qualitative Research Studies in Psychology and Related Fields; GRMS = Guidelines for Reporting Momentary Studies; STROBE = Strengthening the Reporting of Observational Studies in Epidemiology; COREQ = COnsolidated criteria for REporting Qualitative Research; GRAMMS = Good Reporting of a Mixed Methods Study; GCRMR = Guidelines for Conducting and Reporting Mixed Research in the Field of Counselling and Beyond; GREOM = Guidelines for Reporting Evaluations Based on Observational Methodology; SRQR = Standards for Reporting Qualitative Research; MRMM = Methodological Rigor in Mixed Methods; OMS = suitability for assessing studies based on observational methodology; MMS = suitability for assessing mixed-methods studies; QR = suitability for assessing quality of reporting; QD = suitability for assessing quality of design; QM = suitability for assessing quality of measurement; QA = suitability for assessing quality of analysis.
Based on the established criteria, the GREOM was chosen because it was the only tool specific to studies based on observational methodology. Additionally, the MRMM and the MMAT were selected given that they were unique tools that assessed the quality of design, measurement, and analysis.
For the data extraction, Daniel López-Arenas (DLA) and José Mena-Raposo (JMR) were trained to apply the MQSOM, MRMM, GREOM, and MMAT. First, each item and its response options were explained. Salvador Chacón-Moscoso (SCM) mediated if the explanations differed. Then, both coders applied the instruments independently to two papers presenting studies based on observational methodology. Thereafter, DLA and JMR reviewed the coding together with SCM, who intervened in the case of discrepancies. Once the training was complete, DLA and JMR were assigned the same 24 randomly selected papers (25%) to code in order to evaluate the intercoder reliability. After achieving an adequate level of agreement (>.7), DLA applied the instruments to the full sample. Finally, 9 months after the initial coding, DLA re-coded 24 randomly selected papers (25%) to assess the intracoder reliability. The data extraction database is available as Supplementary Material.
Data Analysis
Using Jeffreys’s Amazing Statistics Program (JASP) 0.19.3, the ICC was calculated to evaluate and compare both inter-intracoder concordance of the instruments. Values exceeding .70 were considered adequate (Portney & Walkins, Reference Portney and Watkins2000). Additionally, descriptive statistics were calculated for each dimension. Subsequently, McDonald’s omega (ω) derived through principal factor analysis and Cronbach’s alpha (α) were both employed to assess the reliability based on the internal consistency of each instrument. Results higher than .80 were considered robust reliability evidence, and .65–.80, acceptable (Kalkbrenner, Reference Kalkbrenner2023). For the item discrimination, the average item discrimination index was computed. Results were considered excellent for values higher than .40, adequate for .20–.30, and inadequate for <.20 (Holgado-Tello, Reference Holgado-Tello, Barbero-García, Vila-Abad and Holgado-Tello2015).
Finally, a multitrait-multimethod (MTMM) analysis was conducted to provide evidence of the convergent and discriminant validity of the MQSOM (Campbell & Fiske, Reference Campbell and Fiske1959). This approach is based on the idea that a measurement scale represents the combination of a specific trait and a particular measurement method. The correlation between scores obtained from two measurement instruments may therefore arise from two sources: (1) the fact that they assess the same underlying construct and (2) the similarity of the measurement methods used. To minimize the influence of the method effect, the MTMM approach involves assessing two or more distinct traits using two or more different measurement methods. When the same trait is measured by different methods, strong correlations are expected, providing evidence of convergent validity. Conversely, if different traits show unexpectedly high correlations, this may indicate that methodological similarities are introducing bias, thereby threatening validity. In contrast, when distinct traits are compared, only weak correlations are expected (these constitute discriminant validity evidence). Therefore, Spearman’s correlation (ρ) matrix was constructed with the dimensions from the MQSOM and those from the instruments for comparison included in the analysis using Spearman’s correlation coefficient (ρ) due to the non-normal distribution of the scores. Correlation values over .4 were considered high, over .25 were considered moderate, and over .1 were considered low (Cohen, Reference Cohen1988). For convergence validity, high correlations were expected between dimensions that address similar constructs; conversely, low correlations between dimensions that address different constructs were interpreted as evidence of discriminant validity (Campbell & Fiske, Reference Campbell and Fiske1959).
Results
Inter- and Intracoder Reliability and Descriptive Statistics of the Items
Table 3 presents both inter- and intracoder reliability and descriptive statistics. ICC coefficients were adequate, ranging from .7 to 1. Regarding the descriptive statistics, the median in MQSOM was .67 for the design dimension and .7 for the measurement and analysis dimension; the means were .62 for design and .65 for the measurement and analysis; and the standard deviations were 0.28 for design and 0.18 for measurement and analysis. Regarding MRMM, the median was 1 or .5 for all the dimensions, with 0 being the median only for the dimension of mixed-methods design type. The means ranged from .43 to .90, and the standard deviations were between 0.17 and 0.46.
Descriptive statistics and inter–intracoder reliability of the instruments for comparison
Note. Avg. = average of intraclass correlation coefficients; Min. = minimum; Max. = maximum; M = mean; Mdn = median; SD = standard deviation; S = skewness; K = kurtosis; KS = Kolmogorov–Smirnov normality test; MQSOM design = quality of design; MQSOM M&A = quality of measurement and analysis; MRMM MM design = mixed-methods design type; MRMM writing = elements of writing; GREOM intervention = intervention and expected outcomes.
a ICC is computed for a single item, and there is no minimum or maximum.
ICC and KS obtained p < .05.
Regarding the GREOM, the median was .5 or higher for 66.7% of the dimensions, with a 0 only for design. The means were between .43 and .68, and the standard deviations ranged between 0.21 and 0.46. Regarding the MMAT, the median was .6 for the QUAL and mixed-methods component and .7 for the QUANT component. The means were between .56 and .65, and the standard deviations ranged between .18 and .23. Normal distribution was not found for any of the items of the different instruments.
Reliability Based on Internal Consistency and Discrimination
Table 4 presents the results obtained for the reliability coefficients based on the internal consistency and discrimination indexes of each dimension of the instruments under consideration.
Reliability and discrimination of the dimensions
Note. CI = confidence interval; LL = lower limit; UL = upper limit; Avg. = average of intraclass correlation coefficients; Min. = minimum; Max. = maximum; MQSOM Design = quality of design; MQSOM M&A = quality of measurement and analysis; MRMM MM design = mixed-methods design type; MRMM writing = elements of writing; GREOM intervention = intervention and expected outcomes.
a Reliability coefficients are not computed since the dimension is comprised of a single item.
b The discrimination indexes displayed are not an average but unitary since they are formed by a single item.
The reliability based on the internal consistency of the instruments was adequate only for the MQSOM’s design and the MMAT QUAL and mixed-methods components. Discrimination was considered excellent for all dimensions except MQSOM’s measurement and analysis, and the MRMM’s aims and purpose and data integration, which were considered adequate, and the MRMM’s data collection and data analysis, which were deemed poor.
MTMM Matrix
Table 5 presents the MTMM matrix with Spearman’s correlation coefficients of the MQSOM dimensions with the dimensions of MRMM, GREOM, and MMAT.
Multitrait-multimethod matrix
Note. MQSOM design = quality of design; MRMM5 MM design type = mixed-methods design type.
*p < .05; **p < .01.
The results showed significant correlations of MQSOM with 63.33% of the instrument dimensions (19 correlations), with magnitudes ranging from .21 to .64. Additionally, a significant moderate factor correlation was identified within MQSOM (ρ = .364, p < .001).
Regarding the MRMM checklist, the MQSOM dimension of design exhibited moderate correlations with the dimensions of aims and purpose (ρ = .23, p = .027) and mixed-methods design type (ρ = .22, p = .034) and a strong correlation with elements of writing (ρ = .47, p < .001). Conversely, it showed low and nonsignificant correlations with the dimensions of data collection (ρ = .03, p = .807) and data analysis (ρ = −.05, p = .620). Regarding the GREOM, the MQSOM design dimension exhibited a moderate correlation with the design dimension (ρ = .22, p = .034) and a moderate-to-strong correlation with the intervention and expected outcome dimension (ρ = .34, p = .001). Conversely, it exhibited low and nonsignificant correlations with samples (ρ = −.03, p = .750), data quality control (ρ = .03, p = .812), and results (ρ = −.03, p = .759). Regarding the MMAT checklist, the MQSOM dimension design exhibited a moderate correlation with the QUAL component (ρ = .21, p = .038). Conversely, it exhibited a low and nonsignificant correlation with the QUANT component (ρ = −.04, p = .733).
Regarding the MRMM checklist, the MQSOM dimension measurement and analysis exhibited a moderate correlation with the aims and purpose dimension (ρ = .21, p = .041) and strong correlations with the dimensions data integration (ρ = .48, p < .001), mixed-methods design type (ρ = .45, p < .001), elements of writing (ρ = .42, p < .001), and data analysis (ρ = .61, p < .001). Conversely, it showed low-to-moderate correlation and nonsignificant relationships with the dimensions data collection (ρ = .17, p = .105). Regarding the GREOM, the measurement and analysis dimension exhibited strong correlations with the dimensions design (ρ = .45, p < .001), data quality control (ρ = .42, p < .001), results (ρ = .33, p = .001), intervention and expected outcomes (ρ = .55, p < .001), and instruments (ρ = .61, p < .001). Conversely, it showed a low and nonsignificant correlation with the dimension samples (ρ = .11, p = .276). Regarding the MMAT checklist, the measurement and analysis dimension exhibited a moderate correlation with the QUANT component (ρ = .21, p = .037) and strong correlations with both the QUAL (ρ = .64, p < .001) and mixed-methods (ρ = .64, p < .001) components.
Discussion
The findings provide evidence of adequate inter- and intracoder reliability for the MQSOM and the instruments for comparison. In contrast, both McDonald’s omega and Cronbach’s alpha values indicate a lack of internal consistency for the MQSOM’s measurement and analysis dimension, all the GREOM dimensions, and the MMAT’s QUANT component. Regarding discrimination, the global average discrimination index was deemed acceptable for the MQSOM and all the dimensions of the instruments for comparison, except for the MRMM’s data collection and data analysis. This suggests that these dimensions (which comprise a single item each) may be ineffective in distinguishing between high- and low-quality studies.
The MTMM analysis provides substantial evidence of the convergent and discriminant validity of MQSOM. Specifically, the MQSOM dimensions demonstrated significant correlations with dimensions addressing similar constructs through all instruments for comparison, ranging from moderate to strong. Conversely, the design quality dimension demonstrated low, nonsignificant correlations with the dimensions that assess different constructs (Campbell & Fiske, Reference Campbell and Fiske1959).
Specifically, the MQSOM’s design demonstrates convergent validity evidence with the same constructs from the instrument for comparison, e.g., the MRMM’s and GREOM’s designs. Additionally, it demonstrates convergent validity evidence with several analogous constructs. These include MRMM’s aims and purpose dimension, which involves justifying the use of mixed methods, and its elements of writing dimension, which encompass identifying the study as mixed methods. The MQSOM’s design also shows convergence with GREOM’s intervention and expected outcomes, which requires justification for employing an observational methodology, and with MMAT’s QUAL component. This latter convergence is consistent with the latent construct of design quality addressed by MQSOM, since the initial QUAL phase begins with a justification for the use of observational methodology (Anguera et al., Reference Anguera, Blanco-Villaseñor, Losada and Sánchez-Algarra2020).
The MQSOM’s measurement and analysis demonstrate convergent validity evidence with the same constructs from the instrument for comparison, such as MRMM’s data analysis and GREOM’s instruments, data quality control, and results. Additionally, convergent validity evidence was exhibited with similar constructs, such as MRMM’s data integration and MMAT’s QUAL, QUANT, and mixed-methods components. This finding is congruent with the latent construct quality of measurement and analysis assessed by the MQSOM. In the context of observational methodology, measurement entails the coding of spontaneous behavior to yield a matrix of QUAL data; the integration of data strands is achieved when these QUAL data are transformed into QUANT parameters for analysis, positioning observational methodology as a mixed-methods approach (Anguera et al., Reference Anguera, Blanco-Villaseñor, Losada and Sánchez-Algarra2020).
Regarding discriminant validity evidence, the MQSOM’s design obtained low and nonsignificant correlations with MRMM’s data collection, data integration, and data analysis, GREOM’s samples, instruments, data quality control, and results, and the MMAT’s QUANT and mixed-methods components. Additionally, the MQSOM’s measurement and analysis obtained discriminant evidence with GREOM’s samples.
On the other hand, MQSOM’s measurement and analysis yielded some unexpected results. First, significant correlations were exhibited with MRMM’s both aims and purpose (although low–moderate) and elements of writing (strong). This can be attributed to the pivotal role of the discussion section in the observational methodology process, given that the final step is the QUAL interpretation of QUANT results. The discussion section is also critical in the coding criteria of the MRMM items, which assess the value of mixed-method research for aims and purpose, and in the incorporation of references to mixed-method studies for elements of writing.
Additionally, the MQSOM’s measurement and analysis dimension exhibited significant, strong correlations with the design dimensions of both MRMM and GREOM. This can be attributed to the inherent overlap between research design and research implementation, whereby the quality of the design can influence the quality of subsequent measurement and analysis. Finally, there was a nonsignificant correlation between the MQSOM’s measurement and analysis and the analogous construct, the MRMM’s data collection. This could be attributed to the fact that the MQSOM is centered on methodological quality (e.g., the adequacy of the instrument), while the MRMM focuses on the quality of the reporting (e.g., if the instrument used is specified).
Study Limitations and Further Development
Although this study has contributed to the validity evidence of the MQSOM, several limitations should be considered. First, the internal consistency of MQSOM’s measurement and analysis was below optimal levels. This finding, unexpected based on the results obtained in the original validation of the scale (Sanduvete-Chaves et al., Reference Sanduvete-Chaves, López-Arenas, Anguera and Chacón-Moscoso2025) and based on other applications with a larger sample, indicates a potential need for further research with a larger and more diverse sample of studies based on observational methodology to ensure the reliable measurement of the latent construct. In the present study, we decided to maintain the validity of the dimension in its original version instead of introducing modifications to obtain a higher reliability coefficient, given that the dimension is considered well represented; e.g., each item refers to a different aspect: observation instrument, software, type of parameter, reliability, and type of data analysis.
Additionally, the instruments available for comparison in the mixed methods, such as guidelines and checklists, do not always provide comprehensive psychometric evidence, particularly regarding reliability and construct validity. Furthermore, they focus mainly on the quality of the reporting, while the MQSOM is mainly centered on measuring methodological quality. Consequently, while the evidence supports convergent and discriminant validity evidence, these findings should be interpreted within the context of the differences in the constructs that the instruments compared are measuring. Further research is necessary to compare the MQSOM with other scales that may be developed in the coming years to assess the methodological quality in mixed-methods studies.
Furthermore, the preliminary evidence of convergent and discriminant validity based on correlation analyses should be complemented with additional statistical techniques—such as confirmatory factor analysis, structural equation modeling, or network analysis—to provide a deeper understanding of the latent dimensions in which the MQSOM and the comparison instrument appear (or fail) to converge (Marsh & Grayson, Reference Marsh, Grayson and Hoyle1995). Future studies should also examine the factor invariance of the MQSOM across diverse areas of applied research grounded in observational methodology.
From a practical standpoint, the MQSOM scale can be applied across a wide range of research contexts, such as studies in sports—including soccer (López-Arenas et al., in press), tennis, and basketball. These applications will contribute to synthesizing the accumulated empirical evidence from various lines of research based on observational methodology. Consequently, distinct methodological quality profiles will be available for each research area, facilitating the comparison and extrapolation of high-quality observational research across disciplines.
Conclusions
In summary, this study contributes to an understanding of the psychometric properties of the MQSOM, providing convergent and discriminant validity evidence. The findings indicate high and significant correlations with equal or similar constructs measured with the MQSOM and the instruments for comparison and low and nonsignificant correlations in relation to different constructs. This reinforces the validity evidence of the MQSOM as a useful methodological quality scale.
Supplementary material
The supplementary material for this article can be found at http://doi.org/10.1017/SJP.2026.10021.
Data availability statement
The datasets presented in this study can be found as supplementary material and in the online repository Open Science Framework (https://osf.io/3zbqy/).
Acknowledgements
We would like to thank Wendy Gosselin (American Translation Association member #275293) for reviewing the English version of this text for publication.
Author contribution
S.C.M. and S.S.C conceptualized the study. D.L.A. and J.M.R. investigated the data. D.L.A. and J.M.R. curated the data. D.L.A. is involved in formal analysis. D.L.A. wrote the original draft. S.C.M. and S.S.C. supervised the data. S.S.C. and S.C.M. wrote, reviewed, and edited the manuscript.
Funding statement
This work was supported by ANID, Government of Chile, the National Scientific and Technological Development Fund (FONDECYT) project 1250316 (S.C.M. and S.S.C.); the Julio Olea Grant for Young Researchers of the Spanish Association of Behavioral Science Methodology (AEMCCO; D.L.A.); and the Spanish Ministry of Science, Innovation and Universities (MICIU/AEI/10.13039/501100011033), research project PID2020-115486GB-I00 (all authors). Funding for open access charge: Universidad de Sevilla/CBUA.
Competing interests
None declared.
Open access
