Hostname: page-component-89b8bd64d-z2ts4 Total loading time: 0 Render date: 2026-05-06T12:18:42.607Z Has data issue: false hasContentIssue false
  • English
  • Français

Polytomous Effectiveness Indicators in Complex Problem-Solving Tasks and Their Applications in Developing Measurement Model

Published online by Cambridge University Press:  01 January 2025

Pujue Wang Open the ORCID record for Pujue Wang [Opens in a new window]  and
Hongyun Liu Open the ORCID record for Hongyun Liu [Opens in a new window]
Pujue Wang
Affiliation:
Beijing Normal University Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education (Beijing Normal University)
Hongyun Liu*
Affiliation:
Beijing Normal University Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education (Beijing Normal University)
*
Correspondence should be made to Hongyun Liu, Beijing Key Laboratory of Applied Experimental Psychology, National Demonstration Center for Experimental Psychology Education (Beijing Normal University), Faculty of Psychology, Beijing Normal University, No. 19, Xin Jie Kou Wai St., Hai Dian District, Beijing100875, People’s Republic of China. Email: hyliu@bnu.edu.cn
Get access Rights & Permissions [Opens in a new window]

Abstract

Recent years have witnessed the emergence of measurement models for analyzing action sequences in computer-based problem-solving interactive tasks. The cutting-edge psychometrics process models require pre-specification of the effectiveness of state transitions often simplifying them into dichotomous indicators. However, the dichotomous effectiveness becomes impractical when dealing with complex tasks that involve multiple optimal paths and numerous state transitions. Building on the concept of problem-solving, we introduce polytomous indicators to assess the effectiveness of problem states ds\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$d_{s}$$\end{document} and state-to-state transitions Δds→s′\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$${\mathrm {\Delta }d}_{\mathrm {s\rightarrow s'}}$$\end{document}. The three-step evaluation method for these two types of indicators is proposed and illustrated across two real problem-solving tasks. We further present a novel psychometrics process model, the sequential response model with polytomous effectiveness indicators (SRM-PEI), which is tailored to encompass a broader range of problem-solving tasks. Monte Carlo simulations indicated that SRM-PEI performed well in the estimation of latent ability and transition tendency parameters across different conditions. Empirical studies conducted on two real tasks supported the better fit of SRM-PEI over previous models such as SRM and SRMM, providing rational and interpretable estimates of latent abilities and transition tendencies through effectiveness indicators. The paper concludes by outlining potential avenues for the further application and enhancement of polytomous effectiveness indicators and SRM-PEI.

Keywords

polytomous effectiveness indicatorsSRM-PEIprocess dataproblem-solving

Information

Type
Theory & Methods
Information
Psychometrika , Volume 89 , Issue 3 , September 2024 , pp. 877 - 902
Copyright
Copyright © 2024 The Author(s), under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

Footnotes

Supplementary Information The online version contains supplementary material available at https://doi.org/10.1007/s11336-024-09963-8.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

References

Anderson, J. R., Funke, J., & Plata, G. (Eds.). (2007). Cognitive psychologic (6 Aufl.). Spektrum Akademischer Verlag http://www.gbv.de/dms/bs/toc/529836963.pdf.Google Scholar
Arieli-Attali, M., Ou, L., Simmering, V. R. (2019). Understanding test takers’ choices in a self-adapted test: A hidden Markov modeling of process data. Frontiers in Psychology, 10, 83.CrossRefGoogle Scholar
Bergner, Y., von Davier, A. A. (2019). Process data in NAEP: Past, present, and future. Journal of Educational and Behavioral Statistics, 44(6), 706–732.CrossRefGoogle Scholar
Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29–51.CrossRefGoogle Scholar
Brooks, S. P., Gelman, A. (1998). General methods for monitoring convergence of iterative simulations. Journal of Computational and Graphical Statistics, 7, 434–455.CrossRefGoogle Scholar
Buchner, A., Funke, J. (1993). Finite-state automata: Dynamic task environments in problem-solving research. The Quarterly Journal of Experimental Psychology, 46(1), 83–118.CrossRefGoogle Scholar
Chen, Y. (2020). A continuous-time dynamic choice measurement model for problem-solving process data. Psychometrika, 85(4), 1052–1075.CrossRefGoogle ScholarPubMed
Cormen, T. H., Leiserson, C. E., Rivest, R. L., Stein, C. (2022). Introduction to algorithms, 4Cambridge: MIT Press 563–572.Google Scholar
Fu, Y., Zhan, P., Chen, Q., Jiao, H. (2023). Joint modeling of action sequences and action time in computer-based interactive tasks. Behav Res Methods, .CrossRefGoogle ScholarPubMed
Funke, J. (2001). Dynamic systems as tools for analysing human judgement. Think Reason, 7, 69–89.CrossRefGoogle Scholar
Geisser, S., Eddy, W. F. (1979). A predictive approach to model selection. J Am Stat Assoc, 74, 153–160.CrossRefGoogle Scholar
Gelfand, A. E., Dey, D. K. (1994). Bayesian model choice: Asymptotics and exact calculations. Journal of the Royal Statistical Society Series B, 56, 501–514.CrossRefGoogle Scholar
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., Rubin, D. B. (2014). Bayesian data analysis, 3Boca Raton: Chapman & Hall/CRC Press.Google Scholar
Gelman, A., Meng, X.-L., Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statistica Sinica, 6, 733–760.Google Scholar
Gelman, A., Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science, 7(4), 457–472.CrossRefGoogle Scholar
Griffin, P., Care, E. (2014). Assessment and teaching of 21st century skills: Methods and approach, New York, NY: Springer.Google Scholar
Guttman, I. (1967). The use of the concept of a future observation in goodness-of-fit problems. Journal of the Royal Statistical Society: Series B (Methodological), 29(1), 83–100.CrossRefGoogle Scholar
Han, Y., Liu, H., Ji, F. (2022). A sequential response model for analyzing process data on technology-based problem-solving tasks. Multivariate Behavioral Research, 57(6), 960–977.CrossRefGoogle ScholarPubMed
Han, Y., Wilson, M. (2022). Analyzing student response processes to evaluate success on a technology-based problem-solving task. Applied Measurement in Education, 35(1), 33–45.CrossRefGoogle Scholar
He, Q., von Davier, M. (2015). Identifying feature sequences from process data in problem-solving items with n-grams. In van der Ark, L. A., Bolt, D. M., Wang, W.-C., Douglas, J. A., Chow, S.-M. (Eds), Quantitative psychology research, Berlin: Springer 173–190.CrossRefGoogle Scholar
He, Q., von Davier, M. (2016). Analyzing process data from problem-solving items with N-Grams: Insights from a computer-based large-scale assessment. In Rosen, Y., Ferrara, S., Mosharraf, M. (Eds), Handbook of research on technology tools for real-world skill development, Pennsylvania: IGI Global 750–777.CrossRefGoogle Scholar
Koehler, E., Brown, E., Haneuse, J. P. A. (2009). On the assessment of monte carlo error in simulation-based statistical analyses. The American Statistician, 63(2), 155–162.CrossRefGoogle ScholarPubMed
LaMar, M. M. (2018). Markov decision process measurement model. Psychometrika, 83(1), 67–88.CrossRefGoogle ScholarPubMed
Levy, R., Mislevy, R. J. (2016). Bayesian psychometric modeling, Cambridge: CRC Press.Google Scholar
Li, M., Liu, H., Cai, M., Yuan, J. (2023). Estimation of individuals’ collaborative problem solving ability in computer-based assessment. Education and Information Technologies, .Google Scholar
Liu, H., Liu, Y., Li, M. (2018). Analysis of process data of PISA 2012 computer-based problem solving: Application of the modified multilevel mixture IRT model. Frontiers in Psychology, 9, 1372.CrossRefGoogle ScholarPubMed
Mayer, R. E., Wittrock, M. C. (2006). Problem solving. In Alexander, P. A., Winne, P. H. (Eds), Handbook of educational psychology, 2Mahwah: Erlbaum 287–304.Google Scholar
Newell, A., Simon, H. A. (1972). Human problem solving, Englewood Cliffs: Prentice-Hall.Google Scholar
OECD (2014). PISA 2012 results: Creative problem solving: Students’ skills in tackling real-life problems (Vol. V). OECD.CrossRefGoogle Scholar
OECD. (2016). PISA 2015 Assessment and analytical framework: Science. Reading, mathematic and financial literacy: PISA. OECD Publishing. https://10.1787/9789264255425-en.CrossRefGoogle Scholar
OECD (2018). The future of education and skills: Education 2030, Paris: OECD Publishing.Google Scholar
Patz, R. J., Junker, B. W. (1999). Applications and extensions of MCMC in IRT: Multiple item types, missing data, and rated responses. Journal of Educational and Behavioral Statistics, 24(4), 342–366.CrossRefGoogle Scholar
Patz, R. J., Junker, B. W. (1999). A straightforward approach to Markov chain Monte Carlo methods for item response models. Journal of Educational and Behavioral Statistics, 24(2), 146–178.CrossRefGoogle Scholar
Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applies statistician. The Annals of Statistics, 12, 1151–1172.CrossRefGoogle Scholar
Shu, Z., Bergner, Y., Zhu, M., Hao, J., von Davier, A. A. (2017). An item response theory analysis of problem-solving processes in scenario-based tasks. Psychological Test and Assessment Modeling, 59(1), 109–131.Google Scholar
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., & Van der Linde, A. (1998). Bayesian deviance, the effective number of parameters, and the comparison of arbitrarily complex models. MRC Biostatistics Unit: Technical report.Google Scholar
Tang, X. (2023). A latent hidden Markov model for process data. Psychometrika, .Google ScholarPubMed
Tang, X., Wang, Z., He, Q., Liu, J., Ying, Z. (2020). Latent feature extraction for process data via multidimensional scaling. Psychometrika, 85(2), 378–397.CrossRefGoogle ScholarPubMed
von Davier, A. A. (2017). Computational psychometrics in support of collaborative educational assessments. Journal of Educational Measurement, 54(1), 3–11.CrossRefGoogle Scholar
Xiao, Y., He, Q., Veldkamp, B., Liu, H. (2021). Exploring latent states of problem-solving competence using hidden Markov model on process data. Journal of Computer Assisted Learning, 37(5), 1232–1247.CrossRefGoogle Scholar
Xiao, Y., & Liu, H. (2023). A state response measurement model for problem-solving process data. Behavior Research Methods, 1–20. https://10.3758/s13428-022-02042-9.CrossRefGoogle Scholar
Yuan, J., Xiao, Y., Liu, H. (2019). Assessment of collaborative problem solving based on process stream data: A new paradigm for extracting indicators and modeling dyad data. Frontiers in Psychology, 10, 369.CrossRefGoogle Scholar
Supplementary material: File

Wang and Liu Supplementary material

Wang and Liu Supplementary material
Download Wang and Liu Supplementary material(File)
File 1.1 MB