Hostname: page-component-76d6cb85b7-ntvhh Total loading time: 0 Render date: 2026-07-26T01:19:05.363Z Has data issue: false hasContentIssue false
  • English
  • Français

A Variational Maximization–Maximization Algorithm for Generalized Linear Mixed Models with Crossed Random Effects

Published online by Cambridge University Press:  01 January 2025

Minjeong Jeon ,
Frank Rijmen  and
Sophia Rabe-Hesketh
Minjeong Jeon*
Affiliation:
University of California, Los Angeles
Frank Rijmen
Affiliation:
American Institutes for Research
Sophia Rabe-Hesketh
Affiliation:
University of California, Berkeley
*
Correspondence should be made to Minjeong Jeon, Department of Education, University of California, Los Angeles, 405 Hilgard Avenue, Los Angeles, CA 90095, USA. Email: mjjeon@ucla.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

We present a variational maximization–maximization algorithm for approximate maximum likelihood estimation of generalized linear mixed models with crossed random effects (e.g., item response models with random items, random raters, or random occasion-specific effects). The method is based on a factorized variational approximation of the latent variable distribution given observed variables, which creates a lower bound of the log marginal likelihood. The lower bound is maximized with respect to the factorized distributions as well as model parameters. With the proposed algorithm, a high-dimensional intractable integration is translated into a two-dimensional integration problem. We incorporate an adaptive Gauss–Hermite quadrature method in conjunction with the variational method in order to increase computational efficiency. Numerical studies show that under the small sample size conditions that are considered the proposed algorithm outperforms the Laplace approximation.

Keywords

variational approximationlower boundKullback–Leibler divergenceEM algorithmVMM algorithmadaptive quadratureGLMMcrossed random effects

Information

Type
Original paper
Information
Psychometrika , Volume 82 , Issue 3 , September 2017 , pp. 693 - 716
Copyright
Copyright © 2017 The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Electronic supplementary material The online version of this article (doi:10.1007/s11336-017-9555-z) contains supplementary material, which is available to authorized users.

References

Bates, D., Maechler, M., Bolker, B., & Walker, S. (2014). lme4: Linear mixed-effects models using Eigen and S4. R package version 1.1-6. http://CRAN.R-project.org/package=lme4.Google Scholar
Bates, D. M. (2011). Linear mixed model implementation in lme4. http://cran.rproject.org/web/packages/lme4/vignettes/Implementation.pdf.Google Scholar
Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Andrea, M. H., Chassin, L., & Zucker, R.. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18, 475–493. doi:10.1037/a0032475 3964937.CrossRefGoogle ScholarPubMed
Bickel, P., Choi, D., Chang, X., & Zhang, H.. (2013). Asymptotic normality of maximum likelihood and its variational approximation for stochastic blockmodels. Annals of Statistics, 41, 1922–1943. doi:10.1214/13-AOS1124.CrossRefGoogle Scholar
Bishop, C., Lawrence, N., Jaakkola, T., Jordan, M.Jordan, M., Kearns, M., & Solla, S.. (1998). Approximating posterior distributions in belief networks using mixtures. Advances in neural information processing systems. Cambridge, MA: MIT Press 416–422.Google Scholar
Bock, R. D., & Aitkin, M.. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443–459. doi:10.1007/BF02293801.CrossRefGoogle Scholar
Booth, J., & Hobert, J.. (1999). Maximizing generalized linear mixed model likelihoods with an automated Monte Carlo EM algorithm. Journal of the Royal Statistical Society Series B, 61, 265–285. doi:10.1111/1467-9868.00176.CrossRefGoogle Scholar
Breslow, N., & Clayton, D.. (1993). Approximate inference in generalized linear mixed models. Journal of the American Statistical Association, 88, 9–25.CrossRefGoogle Scholar
Browne, W., & Draper, D.. (2006). A comparison of Bayesian and likelihood methods for fitting multilevel models. Bayesian Analysis, 1, 473–514. doi:10.1214/06-BA117.CrossRefGoogle Scholar
Cai, L.. (2010). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581–612. doi:10.1007/s11336-010-9178-0.CrossRefGoogle Scholar
Cai, L.. (2010). A two-tier full-information item factor analysis model with applications. Psychometrika, 75, 581–612. doi:10.1007/s11336-010-9178-0.CrossRefGoogle Scholar
Cho, S-J, & Rabe-Hesketh, S.. (2011). Alternating imputation posterior estimation of models with crossed random effects. Computational Statistics and Data Analysis, 55, 12–25. doi:10.1016/j.csda.2010.04.015.CrossRefGoogle Scholar
De Boeck, P., & Wilson, M.Explanatory item response models: A generalized linear and nonlinear approach 2004 New York: Springerdoi:10.1007/978-1-4757-3990-9.CrossRefGoogle Scholar
Efron, B.. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7, 1–26. doi:10.1214/aos/1176344552.CrossRefGoogle Scholar
Foley, B. P., (2010). Improving IRT parameter estimates with small sample sizes: Evaluating the efficacy of a new data augmentation technique. Lincoln: University of Nebraska, Lincoln.Google Scholar
Fox, J., & Glas, C. A.. (2001). Bayesian estimation of a multilevel IRT model using Gibbs sampling. Psychometrika, 66, 271–288. doi:10.1007/BF02294839.CrossRefGoogle Scholar
Geweke, J.. (1989). Bayesian inference in econometric models using Monte Carlo integration. Econometrica, 57, 1317–1339. doi:10.2307/1913710.CrossRefGoogle Scholar
Glas, C. A. W., & van der Linden, W. J.. (2003). Computerized adaptive testing with item cloning. Applied Psychological Measurement, 27, 247–261. doi:10.1177/0146621603027004001.CrossRefGoogle Scholar
Goldstein, H.. (1987). Multilevel covariance component models. Biometrika, 74, 430–431. doi:10.1093/biomet/74.2.430.CrossRefGoogle Scholar
Hall, P., Ormerod, J. T., & Wand, M. P.. (2011). Theory of Gaussian variational approximation for a Poisson mixed model. Statistica Sinica, 21, 369–389.Google Scholar
Hall, P., Pham, T., Wand, M. P., & Wang, S. S. J.. (2011). Asymptotic normality and valid inference for Gaussian variational approximation. Annals of Statistics, 39, 2502–2532. doi:10.1214/11-AOS908.CrossRefGoogle Scholar
Humphreys, K., & Titterington, D.. (2003). Variational approximations for categorical causal modeling with latent variables. Psychometrika, 68, 391–412. doi:10.1007/BF02294734.CrossRefGoogle Scholar
Humphreys, K., & Titterington, D.. (2003). Variational approximations for categorical causal modeling with latent variables. Psychometrika, 68, 391–412. doi:10.1007/BF02294734.CrossRefGoogle Scholar
Janssen, R., Schepers, J., Peres, D.De Boeck, P., & Wilson, M.. (2004). Models with item and item group predictors. Explanatory item response models: A generalized linear and nonlinear approach. New York: Springer 189–212. doi:10.1007/978-1-4757-3990-9_6.CrossRefGoogle Scholar
Jeon, M., Rijmen, F., & Rabe-Hesketh, S. (2014). A multitrait-multimethod model with a general factor and interaction effects (in preparation)..Google Scholar
Joe, H.. (2008). Accuracy of Laplace approximation for discrete response mixed models. Computational Statistics and Data Analysis, 52, 5066–5074. doi:10.1016/j.csda.2008.05.002.CrossRefGoogle Scholar
Jordan, M. I.. (2004). Graphical models. Statistical Science, 19, 140–155. doi:10.1214/088342304000000026.CrossRefGoogle Scholar
Kamata, A.. (2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79–93. doi:10.1111/j.1745-3984.2001.tb01117.x.CrossRefGoogle Scholar
Karim, M., & Zeger, S.. (1992). Generalized linear models with random effects: Salamander mating revisited. Biometrics, 48, 631–644. doi:10.2307/2532317.CrossRefGoogle ScholarPubMed
Koehler, E., Brown, E., & Haneuse, SJ-P. (2009). On the assessment of Monte Carlo error in simulation-based statistical analyses. The American Statistician, 63, 155–162. doi:10.1198/tast.2009.0030 3337209.CrossRefGoogle ScholarPubMed
Kullback, S., & Leibler, R. A.. (1951). On information and sufficiency. The Annals of Mathematical Statistics, 22, 79–86. doi:10.1214/aoms/1177729694.CrossRefGoogle Scholar
Lindstrom, M. J., & Bates, D. M.. (1988). Newton–Raphson and EM algorithms for linear mixed-effects models for repeated-measures data. Journal of the American Statistical Association, 83, 1014–1022.Google Scholar
Liu, Q., & Pierce, D. A.. (1994). A note on Gauss–Hermite quadrature. Biometrika, 81, 624–629.Google Scholar
Luts, J., & Ormerod, J.. (2014). Mean field variational Bayesian inference for support vector machine classification. Computational Statistics and Data Analysis, 73, 163–176. doi:10.1016/j.csda.2013.10.030.CrossRefGoogle Scholar
McCullagh, P., & Nelder, J.Generalized Linear Models 1989 New York: Chapman and Halldoi:10.1007/978-1-4899-3242-6.CrossRefGoogle Scholar
McCulloch, C. E.. (1997). Maximum likelihood algorithms for generalized linear mixed models. Journal of the American Statistical Association, 92, 162–170. doi:10.1080/01621459.1997.10473613.CrossRefGoogle Scholar
Neal, R. M., & Hinton, G.. (1998). A view of the EM algorithm that justifies incremental, sparse, and other variants. Learning in Graphical Models. Dordrecht: Kluwer Academic Publishers 355–368. doi:10.1007/978-94-011-5014-9_12.CrossRefGoogle Scholar
Neville, S., Ormerod, J., & Wand, M.. (2014). Mean field variational Bayes for continuous sparse signal shrinkage: Pitfalls and remedies. Electronic Journal of Statistics, 1, 1113–1151. doi:10.1214/14-EJS910.Google Scholar
Ormerod, J.. (2011). Grid based variational approximations. Computational Statistics and Data Analysis, 55, 45–56. doi:10.1016/j.csda.2010.04.024.CrossRefGoogle Scholar
Ormerod, J. T., & Wand, M. P.. (2010). Explaining variational approximations. The American Statistician, 64, 140–153. doi:10.1198/tast.2010.09058.CrossRefGoogle Scholar
Ormerod, J. T., & Wand, M. P.. (2012). Gaussian variational approximate inference for generalized linear mixed models. Journal of Computational and Graphical Statistics, 21, 2–17. doi:10.1198/jcgs.2011.09118.CrossRefGoogle Scholar
Parisi, G.Statistical field theory 1988 Redwood City, CA: Addison-Wesley.Google Scholar
Patz, R. J., Junker, B. W., Johnson, M. S., & Mariano, L. T.. (2002). The hierarchical rater model for rated test items and its application to large-scale educational assessment data. Journal of Educational and Behavioral Statistics, 27, 341–384. doi:10.3102/10769986027004341.CrossRefGoogle Scholar
Pham, T., Ormerod, J., & Wand, M.. (2013). Mean field variational Bayesian inference for nonparametric regression with measurement error. Computational Statistics and Data Analysis, 68, 375–387. doi:10.1016/j.csda.2013.07.014.CrossRefGoogle Scholar
Plummer, M. (2003). Jags: A program for analysis of Bayesian graphical models using Gibbs sampling. http://citeseer.ist.psu.edu/plummer03jags.html.Google Scholar
Rabe-Hesketh, S., Skrondal, A., & Pickles, A.. (2004). Generalized multilevel structural modelling. Psychometrika, 69, 167–190. doi:10.1007/BF02295939.CrossRefGoogle Scholar
Rabe-Hesketh, S., Skrondal, A., & Pickles, A.. (2005). Maximum likelihood estimation of limited and discrete dependent variable models with nested random effects. Journal of Econometrics, 128, 301–323. doi:10.1016/j.jeconom.2004.08.017.CrossRefGoogle Scholar
Rijmen, F., & Jeon, M.. (2013). Fitting an item response theory model with random item effects across groups by a variational approximation method. The Annals of Operations Research, 106, 647–662. doi:10.1007/s10479-012-1181-7.CrossRefGoogle Scholar
Rijmen, F., Jeon, M., Rabe-Hesketh, S., & von Davier, M.. (2014). A third order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 39, 235–256. doi:10.3102/1076998614531045.CrossRefGoogle Scholar
Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P.. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8, 185–205. doi:10.1037/1082-989X.8.2.185.CrossRefGoogle ScholarPubMed
Roose, M., & Held, L.. (2011). Sensitivity analysis in Bayesian generalized linear mixed models for binary data. Bayesian Analysis, 6, 259–278. doi:10.1214/11-BA609.Google Scholar
Saul, L., Jaakkola, T., & Jordan, M.. (1996). Mean field theory for sigmoid belief networks. Journal of Artificial Intelligence Research, 4, 61–76.CrossRefGoogle Scholar
Searle, S. R., Casella, G., & McCulloch, C. E. (1992). Variance Components. New York: Wileydoi:10.1002/9780470316856.CrossRefGoogle Scholar
Skrondal, A., & Rabe-Hesketh, S.. (2009). Prediction in multilevel generalized linear models. Journal of the Royal Statistical Society Series A, 172, 659–687. doi:10.1111/j.1467-985X.2009.00587.x.CrossRefGoogle Scholar
Tan, S. L., & Nott, D. J.. (2014). Variational approximation for mixtures of linear mixed models. Journal of Computational and Graphical Statistics, 23, 564–585. doi:10.1080/10618600.2012.761138.CrossRefGoogle Scholar
Tierney, L., & Kadane, J. B.. (1986). Accurate approximations for posterior moments and densities. Journal of the American Statistical Association, 81, 82–86. doi:10.1080/01621459.1986.10478240.CrossRefGoogle Scholar
Vansteelandt, K. (2000). Formal models for contextualized personality psychology. Unpublished doctoral dissertation, K.U. Leuven, Belgium..Google Scholar
von Davier, M., & Sinharay, S.. (2010). Stochastic approximation methods for latent regression item response models. Journal of Educational and Behavioral Statistics, 35, 174–193. doi:10.3102/1076998609346970.CrossRefGoogle Scholar
Wand, M., Ormerod, J., Padoan, S., & Fruhwirth, R.. (2011). Mean field variational Bayes for elaborate distributions. Bayesian Analysis, 6, 847–900. doi:10.1214/11-BA631.CrossRefGoogle Scholar
Wolfinger, R.. (1993). Laplace’s approximation for nonlinear mixed models. Biometrika, 80, 791–795. doi:10.1093/biomet/80.4.791.CrossRefGoogle Scholar
Zhao, K., & Lian, H.. (2014). Variational inferences for partially linear additive models with variable selection. Computational Statistics & Data Analysis, 80, 223–239. doi:10.1016/j.csda.2014.07.003.CrossRefGoogle Scholar
Supplementary material: File

Jeon et al. supplementary material

Jeon et al. supplementary material
Download Jeon et al. supplementary material(File)
File 72.4 KB