Hostname: page-component-89b8bd64d-dvtzq Total loading time: 0 Render date: 2026-05-06T23:22:55.375Z Has data issue: false hasContentIssue false
  • English
  • Français

Resampling-Based Inference Methods for Comparing Two Coefficients Alpha

Published online by Cambridge University Press:  01 January 2025

Markus Pauly ,
Maria Umlauft  and
Ali Ünlü
Markus Pauly*
Affiliation:
Ulm University
Maria Umlauft
Affiliation:
Ulm University
Ali Ünlü
Affiliation:
Technical University of Munich
*
Correspondence should be made to Markus Pauly, Institute of Statistics, Ulm University, Ulm, Germany. Email: markus.pauly@uni-ulm.de; URL: http://www.uni-ulm.de/mawi/statistics/team/professors/prof-dr-markus-pauly.html
Get access Rights & Permissions [Opens in a new window]

Abstract

The two-sample problem for Cronbach’s coefficient αC\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\alpha _C$$\end{document}, as an estimate of test or composite score reliability, has attracted little attention compared to the extensive treatment of the one-sample case. It is necessary to compare the reliability of a test for different subgroups, for different tests or the short and long forms of a test. In this paper, we study statistical procedures of comparing two coefficients αC,1\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\alpha _{C,1}$$\end{document} and αC,2\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\alpha _{C,2}$$\end{document}. The null hypothesis of interest is H0:αC,1=αC,2\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$H_0 : \alpha _{C,1} = \alpha _{C,2}$$\end{document}, which we test against one-or two-sided alternatives. For this purpose, resampling-based permutation and bootstrap tests are proposed for two-group multivariate non-normal models under the general asymptotically distribution-free (ADF) setting. These statistical tests ensure a better control of the type-I error, in finite or very small sample sizes, when the state-of-affairs ADF large-sample test may fail to properly attain the nominal significance level. By proper choice of a studentized test statistic, the resampling tests are modified in order to be valid asymptotically even in non-exchangeable data frameworks. Moreover, extensions of this approach to other designs and reliability measures are discussed as well. Finally, the usefulness of the proposed resampling-based testing strategies is demonstrated in an extensive simulation study and illustrated by real data applications.

Keywords

bootstrapcoefficient alphaCronbach’s alphanon-normalitypermutationreliabilityresampling-based inference

Information

Type
Original Paper
Information
Psychometrika , Volume 83 , Issue 1 , March 2018 , pp. 203 - 222
Copyright
Copyright © 2017 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

The work of Markus Pauly and Maria Umlauft was supported by the German Research Foundation project DFG-PA 2409/3-1.

Electronic supplementary material The online version of this article (https://doi.org/10.1007/s11336-017-9601-x) contains supplementary material, which is available to authorized users.

References

Bader, K., Hänny, C., Schäfer, V., Neuckel, A., Kuhl, C., (2009). Childhood Trauma Questionnaire—Psychometrische Eigenschaften einer deutschsprachigen Version, Zeitschrift für Klinische Psychologie und Psychotherapie, 38(4) 223–230.CrossRefGoogle Scholar
Bentler, P., (2009). Alpha, dimension-free, and model-based internal consistency reliability, Psychometrika, 74(1) 1372786226.CrossRefGoogle ScholarPubMed
Bernstein, D.P., etal (2003). Development and validation of a brief screening version of the childhood trauma questionnaire, Child Abuse & Neglect, 27(2), 169–190.CrossRefGoogle ScholarPubMed
Bonett, D.G., (2003). Sample size requirements for comparing two alpha coefficients, Applied Psychological Measurement, 27(1) 72–74.CrossRefGoogle Scholar
Bonett, D.G., Wright, T.A., (2015). Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning, Journal of Organizational Behavior, 36(1) 3–15.CrossRefGoogle Scholar
Bradbury, I., (1987). Analysis of variance versus randomization tests—A comparison, British Journal of Mathematical and Statistical Psychology, 40(2) 177–187.CrossRefGoogle Scholar
Brennan, R.L., (2001). Generalizability Theory, New York: Springer.CrossRefGoogle Scholar
Chung, E., Romano, J.P., (2013). Exact and asymptotically robust permutation tests, The Annals of Statistics, 41(2) 484–507.CrossRefGoogle Scholar
Cortina, J.M., (1993). What is coefficient alpha? An examination of theory and applications, Journal of Applied Psychology, 78(1) 98.CrossRefGoogle Scholar
Crocker, L., Algina, J., (1986). Introduction to classical and modern test theory, Princeton: ERIC.Google Scholar
Cronbach, L.J., (1951). Coefficient alpha and the internal structure of tests, Psychometrika, 16(3) 297–334.CrossRefGoogle Scholar
D’Zurilla, T.J., Nezu, A.M., Maydeu-Olivares, A., (2002). Social problem-solving inventory—Revised (SPSI-R), North Tonawanda: Multi Health Systems.Google Scholar
Furr, R.M., Bacharach, V.R., (2013). Psychometrics: An introduction, Thousand Oaks: Sage.Google Scholar
Green, S.B., Yang, Y., (2009). Commentary on coefficient alpha: A cautionary tale, Psychometrika, 74(1) 121–135.CrossRefGoogle Scholar
Green, S.B., Yang, Y., (2009). Reliability on summed item scores using structural equation modeling: An alternative to coefficient alpha, Psychometrika, 74(1) 155–167.CrossRefGoogle Scholar
Gulliksen, H., (2013). Theory of mental tests, Abingdon: Routledge.CrossRefGoogle Scholar
Guttmann, L., (1945). A basis for analyzing test-retest reliability, Psychometrika, 10(4) 255–282.CrossRefGoogle Scholar
Hogan, T.P., Benjamin, A., Brezinski, K.L., (2000). Reliability methods: A note on the frequency of use of various types, Educational and Psychological Measurement, 60(4) 523–531.CrossRefGoogle Scholar
Janssen, A., (1997). Studentized permutation tests for non-iid hypotheses and the generalized Behrens–Fisher problem, Statistics & Probability Letters, 36(1) 9–21.CrossRefGoogle Scholar
Kim, S., Feldt, L.S., (2008). A comparison of tests for equality of two or more independent alpha coefficients, Journal of Educational Measurement, 45(2) 179–193.CrossRefGoogle Scholar
Konietschke, F., Bathke, A.C., Harrar, S.W., Pauly, M., (2015). Parametric and non-parametric bootstrap methods for general MANOVA, Journal of Multivariate Analysis, 140, 291–301.CrossRefGoogle Scholar
Konietschke, F., Pauly, M., (2014). Bootstrapping and permuting paired t-test type statistics, Statistics and Computing, 24(3) 283–296.CrossRefGoogle Scholar
Kuijpers, R.E., Ark, L.A., Croon, M.A., (2013). Testing hypotheses involving Cronbach’s alpha using marginal models, British Journal of Mathematical and Statistical Psychology, 66(3) 503–520.CrossRefGoogle ScholarPubMed
Lord, F.M., Novick, M.R., (1968). Statistical theories of mental test scores, Reading: Addison-Wesley.Google Scholar
Maydeu-Olivares, A., Coffman, D.L., Hartmann, W.M., (2007). Asymptotically distribution-free (ADF) interval estimation of coefficient alpha, Psychological Methods, 12(2) 157.CrossRefGoogle ScholarPubMed
Maydeu-Olivares, A., Coffman, D.L., García-Forero, C., Gallardo-Pujol, D., (2010). Hypothesis testing for coefficient alpha: An SEM approach, Behavior research methods, 42(2) 618–6253027072.CrossRefGoogle ScholarPubMed
McDonald, R.P., (1978). Generalizability in factorable domains: domain validity and generalizability 1, Educational and Psychological Measurement, 38(1) 75–79.CrossRefGoogle Scholar
McDonald, R.P., Test theory: A unified treatment 1999 New York: Psychology Press.Google Scholar
Mellenbergh, G.J., (1996). Measurement precision in test score and item response models, Psychological Methods, 1(3) 293.CrossRefGoogle Scholar
Molenaar, I.W., Sijtsma, K., (1988). Mokken’s approach to reliability estimation extended to multicategory items, Kwantitatieve Methoden, 9(28) 115–126.Google Scholar
Muthén, B., Kaplan, D., (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables, British Journal of Mathematical and Statistical Psychology, 38(2) 171–189.CrossRefGoogle Scholar
Muthén, B., Kaplan, D., (1992). A comparison of some methodologies for the factor analysis of non-normal Likert variables: A note on the size of the model, British Journal of Mathematical and Statistical Psychology, 45(1) 19–30.CrossRefGoogle Scholar
Novick, M.R., Lewis, C., (1966). Coefficient alpha and the reliability of composite measurements, Psychometrika, 32(1) 1–13.CrossRefGoogle Scholar
Nunnally, J.C., Bernstein, I.H., (1978). Psychometric theory, New York: McGraw-Hill.Google Scholar
Omelka, M., Pauly, M., (2012). Testing equality of correlation coefficients in two populations via permutation methods, Journal of Statistical Planning and Inference, 142(6) 1396–1406.CrossRefGoogle Scholar
Padilla, M.A., Divers, J., Newton, M., (2012). Coefficient alpha bootstrap confidence interval under nonnormality, Applied Psychological Measurement, 36(5) 331–348.CrossRefGoogle Scholar
Pauly, M., (2011). Discussion about the quality of F-ratio resampling tests for comparing variances, Test, 20(1) 163–179.CrossRefGoogle Scholar
Pauly, M., Brunner, E., Konietschke, F., (2015). Asymptotic permutation tests in general factorial designs, Journal of the Royal Statistical Society: Series B (Statistical Methodology) 77(2) 461–473.CrossRefGoogle Scholar
Peterson, R.A., (1994). A meta-analysis of Cronbach’s coefficient alpha, Journal of Consumer Research, 21(2) 381–391.CrossRefGoogle Scholar
Prelog, A.J., Berry, K.J., Mielke, PW Jr, (2009). Resampling permutation probability values for Cronbach’s alpha, Perceptual and Motor Skills, 108(2) 431–438.CrossRefGoogle ScholarPubMed
Core, R Team (2016). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria.Google Scholar
Rao, C.R., Sinharay, S., (2006). Handbook of statistics: Psychometrics, New York: Elsevier.Google Scholar
Revelle, W., Zinbarg, R.E., (2009). Coefficient alpha, beta, omega, and the glb: Comments on Sijtsma, Psychometrika, 74(1) 145–154.CrossRefGoogle Scholar
Schury, K., Zimmermann, J., Umlauft, M., Hulbert, A.L., Guendel, H., Ziegenhain, U., Kolassa, I.T., (2017). Childhood maltreatment, postnatal distress and the protective role of social support, Child Abuse & Neglect, 67, 228–239.CrossRefGoogle ScholarPubMed
Sijtsma, K., (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha, Psychometrika, 74(1) 107–120.CrossRefGoogle ScholarPubMed
Sijtsma, K., (2009). Reliability beyond theory and into practice, Psychometrika, 74(1) 169–173.CrossRefGoogle ScholarPubMed
Sijtsma, K., Molenaar, I.W., (1987). Reliability of test score in nonparametric item response theory, Psychometrika, 52, 79–97.CrossRefGoogle Scholar
Still, A., White, A., (1981). The approximate randomization test as an alternative to the F-test in analysis of variance, British Journal of Mathematical and Statistical Psychology, 34(2) 243–252.CrossRefGoogle Scholar
Ten Berge, J.M., Socčan, G., (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality, Psychometrika, 69(4) 613–625.CrossRefGoogle Scholar
Umlauft, M., Konietschke, F., Pauly, M., (2017). Rank-based permutation approaches for nonparametric factorial designs, British Journal of Mathematical and Statistical Psychology, 70(3) 368–390.CrossRefGoogle ScholarPubMed
van der Ark, L.A., (2012). New developments in Mokken scale analysis in R, Journal of Statistical Software, 48(5) 1–27.Google Scholar
van der Ark, L.A., van der Palm, D.W., Sijtsma, K., (2011). A latent class approach to estimating test-score reliability, Applied Psychological Measurement, 35, 380–392.CrossRefGoogle Scholar
van Zyl, J.M., Neudecker, H., Nel, D., (2000). On the distribution of the maximum likelihood estimator of Cronbach’s alpha, Psychometrika, 65(3) 271–280.CrossRefGoogle Scholar
Yuan, K-HGuarnaccia, C.A., Hayslip, B., (2003). A study of the distribution of sample coefficient alpha with the Hopkins symptom checklist: Bootstrap versus asymptotics, Educational and Psychological Measurement, 63(1) 5–23.CrossRefGoogle Scholar
Zinbarg, R.E., Revelle, W., Yovel, I., Li, W., (2005). Cronbachs α\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document}, Revelles β\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta $$\end{document}, and McDonalds ωh\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\omega _h$$\end{document}: Their relations with each other and two alternative conceptualizations of reliability. Psychometrika, 70(1) 123–133.Google Scholar
Supplementary material: File

Pauly et al. supplementary material

Pauly et al. supplementary material 1
Download Pauly et al. supplementary material(File)
File 213.9 KB
Supplementary material: File

Pauly et al. supplementary material

Pauly et al. supplementary material 2
Download Pauly et al. supplementary material(File)
File 4.8 KB
Supplementary material: File

Pauly et al. supplementary material

Pauly et al. supplementary material 3
Download Pauly et al. supplementary material(File)
File 4.7 KB
Supplementary material: File

Pauly et al. supplementary material

Pauly et al. supplementary material 4
Download Pauly et al. supplementary material(File)
File 4.7 KB
Supplementary material: File

Pauly et al. supplementary material 5

Pauly et al. supplementary material 5
Download Pauly et al. supplementary material 5(File)
File 4.7 KB