Hostname: page-component-89b8bd64d-46n74 Total loading time: 0 Render date: 2026-05-07T05:25:01.270Z Has data issue: false hasContentIssue false
  • English
  • Français

A Modeling Framework to Examine Psychological Processes Underlying Ordinal Responses and Response Times of Psychometric Data

Published online by Cambridge University Press:  01 January 2025

Inhan Kang Open the ORCID record for Inhan Kang [Opens in a new window] ,
Dylan Molenaar  and
Roger Ratcliff
Inhan Kang*
Affiliation:
Yonsei University
Dylan Molenaar
Affiliation:
University of Amsterdam
Roger Ratcliff
Affiliation:
The Ohio State University
*
Correspondence should be made to Inhan Kang, Yonsei University, 403 Widang Hall, 50 Yonsei-ro, Seodaemun-gu, Seoul 03722, Republic of Korea. Email: qpsy@yonsei.ac.kr
Get access Rights & Permissions [Opens in a new window]

Abstract

This article presents a joint modeling framework of ordinal responses and response times (RTs) for the measurement of latent traits. We integrate cognitive theories of decision-making and confidence judgments with psychometric theories to model individual-level measurement processes. The model development starts with the sequential sampling framework which assumes that when an item is presented, a respondent accumulates noisy evidence over time to respond to the item. Several cognitive and psychometric theories are reviewed and integrated, leading us to three psychometric process models with different representations of the cognitive processes underlying the measurement. We provide simulation studies that examine parameter recovery and show the relationships between latent variables and data distributions. We further test the proposed models with empirical data measuring three traits related to motivation. The results show that all three models provide reasonably good descriptions of observed response proportions and RT distributions. Also, different traits favor different process models, which implies that psychological measurement processes may have heterogeneous structures across traits. Our process of model building and examination illustrates how cognitive theories can be incorporated into psychometric model development to shed light on the measurement process, which has had little attention in traditional psychometric models.

Keywords

response timepsychological processmeasurementpsychometric process modelingdecision-makingconfidence judgments

Information

Type
Theory & Methods
Information
Psychometrika , Volume 88 , Issue 3 , September 2023 , pp. 940 - 974
Copyright
Copyright © 2023 The Author(s) under exclusive licence to The Psychometric Society

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

Akaike, H.(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6),716–723.CrossRefGoogle Scholar
Andrich, D.(1978). Application of a psychometric rating model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2(4),581–594.CrossRefGoogle Scholar
Andrich, D.(1978). A rating formulation for ordered response categories. Psychometrika, 43, 561–573.CrossRefGoogle Scholar
Baranski, J., &Petrusic, W.(1998). Probing the locus of confidence judgments: experiments on the time to determine confidence. Journal of Experimental Psychology. Human Perception and Performance, 24(3),929–945.CrossRefGoogle ScholarPubMed
Basso, M. A., & Wurtz, R. H.(1998). Modulation of neuronal activity in superior colliculus by changes in target probability. Journal of Neuroscience, 18(18),7519–7534.CrossRefGoogle ScholarPubMed
Beck, J. M., Ma, W. J., Kiani, R., Hanks, T., Churchland, A. K., Roitman, J., Shadlen, M. N., Latham, P. E., &Pouget, A.(2008). Probabilistic population codes for Bayesian decision making. Neuron, 60(6),1142–1152.CrossRefGoogle ScholarPubMed
Bock, R. D., &Jones, L. V.(1968). The measurement and prediction of judgment and choice, Holden-Day.Google Scholar
Bollen, K., &Barb, K. H.(1981). Pearson’s r and coarsely categorized measures. American Sociological Review, 46(2),232–239.CrossRefGoogle Scholar
Bolsinova, M., De Boeck, P., &Tijmstra, J.(2017). Modelling conditional dependence between response and accuracy. Psychometrika, 82(4),1126–1148.CrossRefGoogle ScholarPubMed
Bolsinova, M., &Molenaar, D.(2018). Modeling nonlinear conditional dependence between response time and accuracy. Frontiers in Psychology, 9(1525),1–12.CrossRefGoogle ScholarPubMed
Bolsinova, M., &Molenaar, D.(2019). Nonlinear indicator-level moderation in latent variable models. Multivariate Behavioral Research, 54(1),62–84.CrossRefGoogle ScholarPubMed
Bolsinova, M., &Tijmstra, J.(2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71(1),13–38.CrossRefGoogle ScholarPubMed
Bolsinova, M., Tijmstra, J., &Molenaar, D.(2017). Response moderation models for conditional dependence between response time and response accuracy. British Journal of Mathematical and Statistical Psychology, 70, 257–279.CrossRefGoogle ScholarPubMed
Bolsinova, M., Tijmstra, J., Molenaar, D., &De Boeck, P.(2017). Conditional dependence between response time and accuracy: An overview of its possible sources and directions for distinguishing between them. Frontiers in Psychology, 8, 202.CrossRefGoogle ScholarPubMed
Borsboom, D., Mellenbergh, G. J., &van Heerden, J.(2003). The theoretical status of latent variables. Psychological Review, 110(2),203–219.CrossRefGoogle ScholarPubMed
Borsboom, D., Mellenbergh, G. J., &van Heerden, J.(2004). The concept of validity. Psychological Review, 1114, 1061–1071.CrossRefGoogle Scholar
Brown, S. D., &Heathcote, A.(2008). The simplest complete model of choice response time: Linear ballistic accumulation. Cognitive Psychology, 57(3),153–178.CrossRefGoogle ScholarPubMed
Cowell, R. A., Bussey, T. J., &Saksida, L. M.(2006). Why does brain damage impair memory? a connectionist model of object recognition memory in perirhinal cortex. Journal of Neuroscience, 26(47),12186–12197.CrossRefGoogle ScholarPubMed
Cox, D., &Miller, H. D.(1965). The theory of stochastic processes, Methuen.Google Scholar
De Boeck, P., &Jeon, M.(2019). An overview of models for response times and processes in cognitive tests. Frontiers in Psychology, 10, 102.CrossRefGoogle ScholarPubMed
DiTrapani, J., Jeon, M., De Boeck, P., & Partchev, I.(2016). Attempting to differentiate fast and slow intelligence: Using generalized item response trees to examine the role of speed on intelligence tests. Intelligence, 56, 82–92.CrossRefGoogle Scholar
Embretson, S., & Reise, S. (2000). Item response theory for psychologists. L. Erlbaum Associates.Google Scholar
Fengler, A., Govindarajan, L. N., Chen, T., &Frank, M. J.(2021). Likelihood approximation networks (lans) for fast inference of simulation models in cognitive neuroscience. eLife, 10, e65074.CrossRefGoogle ScholarPubMed
Ferrando, P. J., & Lorenzo-Seva, U.(2007). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 316, 525–543.CrossRefGoogle Scholar
Ferrando, P. J., &Lorenzo-Seva, U.(2007). A measurement model for likert responses that incorporates response time. Multivariate Behavioral Research, 424, 675–706.CrossRefGoogle Scholar
Festinger, L.(1943). Studies in decision: I. decision-time, relative frequency of judgment and subjective confidence as related to physical stimulus difference. Journal of Experimental Psychology, 32(4),291–306.CrossRefGoogle Scholar
Festinger, L.(1943). Studies in decision. ii. an empirical test of a quantitative theory of decision. Journal of Experimental Psychology, 32(5),411–423.CrossRefGoogle Scholar
Forstmann, B., Ratcliff, R., &Wagenmakers, E.-J.(2016). Sequential sampling models in cognitive neuroscience: Advantages, applications, and extensions. Annual Review of Psychology, 67(1),641–666.CrossRefGoogle ScholarPubMed
Gelman, A.(1996). Inference and monitoring convergence.Gilks, W. R., Richardson, S., &Spiegelhalter, D. J. Markov chain monte Carlo in practice, CRC Press 131–143.Google Scholar
Gelman, A., Carlin, J. B., Stern, H. S., &Rubin, D. B.(2013). Bayesian data analysis, 3CRC Press.CrossRefGoogle Scholar
Green, D. M., &Swets, J. A.(1966). Signal detection theory and psychophysics, Wiley.Google Scholar
Hermans, H. J. M(1968). Handleiding bij de prestatie motivatie test [manual of the performance motivation test], Harcourt Assessment B.V.Google Scholar
Hermans, H. J. M., Ter Laak, J. J. F., &Maes, P. C. J. M.(1972). Achievement motivation and fear of failure in family and school. Developmental Psychology, 6, 520–528.CrossRefGoogle Scholar
Jazayeri, M., &Movshon, J.(2006). Optimal representation of sensory information by neural populations. Nature Neuroscience, 9, 690–696.CrossRefGoogle ScholarPubMed
Kang, I., De Boeck, P., &Partchev, I.(2022). A randomness perspective on intelligence processes. Intelligence, 91.CrossRefGoogle Scholar
Kang, I., De Boeck, P., &Ratcliff, R.(2022). Modeling conditional dependence of response accuracy and response time with the diffusion item response theory model. Psychometrika, Advance Online Publication.,Google ScholarPubMed
Kang, I., &Ratcliff, R.(2020). Modeling the interaction of numerosity and perceptual variables with the diffusion model. Cognitive Psychology, 120, 1–42.CrossRefGoogle ScholarPubMed
Kang, I., Ratcliff, R., &Voskuilen, C.(2020). A note on decomposition of sources of variability in perceptual decision-making. Journal of Mathematical Psychology, 98, 102431.CrossRefGoogle ScholarPubMed
Kuiper, N. A.(1981). Convergent evidence for the self as a prototype: The “inverted-u rt effect” for self and other judgments. Personality and Social Psychology Bulletin, 7(3),438–443.CrossRefGoogle Scholar
Kuncel, R. B.(1973). Response processes and relative location of subject and item. Educational and Psychological Measurement, 333, 545–563.CrossRefGoogle Scholar
Lu, J., Wang, C., & Shi, N. (2021). A mixture response time process model for aberrant behaviors and item nonresponses. Multivariate Behavioral Research, Advance Online Publication. https://10.1080/00273171.2021.1948815.CrossRefGoogle Scholar
Luce, R. D.(1986). Response times: Their role in inferring elementary mental organization, Oxford University Publication.Google Scholar
Macmillan, N. A., &Creelman, C. D.(1966). Detection theory: A user’s guide, Taylor & Francis.Google Scholar
McKoon, G., &Ratcliff, R.(2016). Adults with poor reading skills: How lexical knowledge interacts with scores on standardized reading comprehension tests. Cognition, 146, 453–469.CrossRefGoogle ScholarPubMed
McKoon, G., &Ratcliff, R.(2017). Adults with poor reading skills and the inferences they make during reading. Scientific Studies of Reading, 21(4),292–309.CrossRefGoogle Scholar
McKoon, G., &Ratcliff, R.(2018). Adults with poor reading skills, older adults, and college students: The meanings they understand during reading using a diffusion model analysis. Journal of Memory and Language, 102, 115–129.CrossRefGoogle ScholarPubMed
Merkle, E., & Van Zandt, T. (2006). An application of the poisson race model to confidence calibration. Journal of Experimental Psychology. General, 135, 391–408.Google Scholar
Modick, H. E.(1977). A 3-scale measure of achievement motivation: Report on a German extension of the prestatie motivatie test. Diagnostica, 23(4),298–321.Google Scholar
Molenaar, D., &De Boeck, P.(2018). Response mixture modeling: Accounting for heterogeneity in item characteristics across response times. Psychometrika, 83(2),279–297.CrossRefGoogle ScholarPubMed
Molenaar, D., Oberski, D., Vermunt, J., & Boeck, P. D.(2016). Hidden Markov item response theory models for responses and response times. Multivariate Behavioral Research, 51(5),606–626.CrossRefGoogle ScholarPubMed
Molenaar, D., Tuerlinckx, F., &van der Maas, H. L. J.(2015). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioral Research, 50(1),56–74.CrossRefGoogle Scholar
Molenaar, D., Tuerlinckx, F., &van der Maas, H. L. J.(2015). Fitting diffusion item response theory models for responses and response times using the r package diffirt. Journal of Statistical Software, 66(4),1–34.CrossRefGoogle Scholar
Muraki, E.(1990). Fitting a polytomous item response model to likert-type data. Applied Psychological Measurement, 14(1),59–71.CrossRefGoogle Scholar
Muthén, B.(1983). Latent variable structural equation modeling with categorical data. Journal of Econometrics, 22(1),43–65.CrossRefGoogle Scholar
Muthén, B.(1984). A general structural equation model with dichotomous, ordered categorical, and continuous latent variable indicators. Psychometrika, 49, 115–132.CrossRefGoogle Scholar
Olsson, U.(1979). Maximum likelihood estimation of the polychoric correlation coefficient. Psychometrika, 44(4),443–460.CrossRefGoogle Scholar
Partchev, I., &De Boeck, P.(2012). Can fast and slow intelligence be differentiated?. Intelligence, 40(1),23–32.CrossRefGoogle Scholar
Pearson, K.(1901). Mathematical contributions to the theory of evolution. viii. on the inheritance of characters not capable of exact quantitative measurement. Philosophical Transactions of the Royal Society of London A, 195, 79–150.CrossRefGoogle Scholar
Pleskac, T. J., &Busemeyer, J.(2010). Two-stage dynamic signal detection: a theory of choice, decision time, and confidence. Psychological Review, 117(3),864–901.CrossRefGoogle ScholarPubMed
Ranger, J., &Kuhn, J. -T.(2018). Modeling responses and response times in rating scales with the linear ballistic accumulator. Methodology, 14(3),119–132.CrossRefGoogle Scholar
Ranger, J., Kuhn, J. -T., &Szardenings, C.(2017). Analysing model fit of psychometric process models: An overview, a new test and an application to the diffusion model. British Journal of Mathematical and Statistical Psychology, 70(2),209–224.CrossRefGoogle Scholar
Ratcliff, R.(1978). A theory of memory retrieval. Psychological Review, 85(2),59–108.CrossRefGoogle Scholar
Ratcliff, R.(2002). A diffusion model account of response time and accuracy in a brightness discrimination task: Fitting real data and failing to fit fake but plausible data. Psychological Science, 9(2),278–291.Google Scholar
Ratcliff, R.(2018). Decision making on spatially continuous scales. Psychological Review, 125, 888–935.CrossRefGoogle ScholarPubMed
Ratcliff, R., Gomez, P., &McKoon, G.(2003). A diffusion model account of the lexical decision task. Psychological Review, 111(1),159–182.CrossRefGoogle Scholar
Ratcliff, R., Hasegawa, Y., Hasegawa, R., Smith, P., &Segraves, M.(2007). Dual diffusion model for single-cell recording data from the superior colliculus in a brightness-discrimination task. Journal of Neurophysiology, 97, 1756–1774.CrossRefGoogle Scholar
Ratcliff, R., Hasegawa, Y. T., Hasegawa, R. P., Smith, P. L., &Segraves, M. A.(2007). Dual diffusion model for single-cell recording data from the superior colliculus in a brightness-discrimination task. Journal of Neurophysiology, 97(2),1756–1774.CrossRefGoogle Scholar
Ratcliff, R., &Kang, I.(2021). Qualitative speed-accuracy tradeoff effects can be explained by a diffusion/fast-guess mixture model. Scientific Reports, 11, 15169.CrossRefGoogle ScholarPubMed
Ratcliff, R., &McKoon, G.(2008). The diffusion decision model: Theory and data for two-choice decision tasks. Neural Computation, 20(4),873–922.CrossRefGoogle ScholarPubMed
Ratcliff, R.,& Smith, P.(2004). A comparison of sequential sampling models for two-choice reaction time. Psychological Review, 111, 333–367.CrossRefGoogle ScholarPubMed
Ratcliff, R., &Starns, J. J.(2009). Modeling confidence and response time in recognition memory. Psychological Review, 116(1),59–83.CrossRefGoogle ScholarPubMed
Ratcliff, R., &Starns, J. J.(2013). Modeling confidence judgments, response times, and multiple choices in decision making: Recognition memory and motion discrimination. Psychological Review, 120(3),697–719.CrossRefGoogle ScholarPubMed
Ratcliff, R., Voskuilen, C., &McKoon, G.(2018). Internal and external sources of variability in perceptual decision-making. Psychological Review, 125(1),33–46.CrossRefGoogle ScholarPubMed
Rouder, J., Province, J., Morey, R., Gómez, P., &Heathcote, A.(2015). The lognormal race: A cognitive-process model of choice and latency with desirable psychometric properties. Psychometrika, 80(2),491–513.CrossRefGoogle Scholar
Samejima, F.(1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34(4, Pt. 2),100.Google Scholar
Samejima, F.(1997). Graded response model.van der Linden, W. J., &Hambleton, R. Handbook of modern item response theory, Springer 85–100.CrossRefGoogle Scholar
Schnipke, D. L., &Scrams, D. J.(1997). Modeling item response times with a two-state mixture model: A new method of measuring speededness. Journal of Educational Measurement, 34(3),213–232.CrossRefGoogle Scholar
Schwarz, G.(1978). Estimating the dimension of a model. The Annals of Statistics, 6(2),461–464.CrossRefGoogle Scholar
Skrondal, A., &Rabe-Hesketh, S.(2004). Generalized latent variable modeling: Multilevel, longitudinal, and structural equation models, Chapman & HallCRC.CrossRefGoogle Scholar
Smith, P. L.(2000). Stochastic dynamic models of response time and accuracy: A foundational primer. Journal of Mathematical Psychology, 44(3),408–463.CrossRefGoogle ScholarPubMed
Smith, P. L., &Vickers, D.(1988). The accumulator model of two-choice discrimination. Journal of Mathematical Psychology, 32(2),135–168.CrossRefGoogle Scholar
Spiegelhalter, D. J., Best, N. G., Carlin, B. P., &Van Der Linde, A.(2002). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64(4),583–639.CrossRefGoogle Scholar
Stan Development Team. (2021). Stan modeling language user’s guide and reference manual stan modeling language user’s guide and reference manual. Retrieved from http://mc-stan.org/.Google Scholar
Takane, Y., &De Leeuw, J.(1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52, 393–408.CrossRefGoogle Scholar
Thurstone, L. L.(1927). A law of comparative judgment. Psychological Review, 34(4),273–286.CrossRefGoogle Scholar
Thurstone, L. L.(1927). Psychophysical analysis. The American Journal of Psychology, 38(3),368–389.CrossRefGoogle Scholar
Thurstone, L. L.(1928). Attitudes can be measured. American Journal of Sociology, 33, 529–554.CrossRefGoogle Scholar
Torgenson, W. S.(1958). Theory and methods of scaling, Wiley.Google Scholar
Tuerlinckx, F., &De Boeck, P.(2005). Two interpretations of the discrimination parameter. Psychometrika, 70(4),629–650.CrossRefGoogle Scholar
Tuerlinckx, F., Molenaar, D., & van der Maas, H. L. J.(2016). Diffusion-based response-time models.van der Linden, W. J. Handbook of item response theory, Chapman and Hall/CRC 283–300.Google Scholar
Turner, B. M., &Sederberg, P. B.(2012). Approximate Bayesian computation with differential evolution. Journal of Mathematical Psychology, 56(5),375–385.CrossRefGoogle Scholar
Turner, B. M., &Sederberg, P. B.(2014). A generalized, likelihood-free method for posterior estimation. Psychonomic Bulletin and Review, 21, 227–250.CrossRefGoogle ScholarPubMed
Turner, B. M., &Van Zandt, T.(2012). A tutorial on approximate Bayesian computation. Journal of Mathematical Psychology, 56(2),69–85.CrossRefGoogle Scholar
Turner, B. M., &Van Zandt, T.(2014). Hierarchical approximate Bayesian computation. Psychometrika, 79, 185–209.CrossRefGoogle ScholarPubMed
Usher, M., &McClelland, J.(2001). The time course of perceptual choice: the leaky, competing accumulator model. Psychological Review, 108, 550–592.CrossRefGoogle ScholarPubMed
van der Linden, W. J.(2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72(3),287–308.CrossRefGoogle Scholar
van der Maas, H. L. J., Molenaar, D., Maris, G., Kievit, R. A., &Borsboom, D.(2011). Cognitive psychology meets psychometric theory: On the relation between process models for decision making and latent variable models for individual differences. Psychological Review, 118(2),339–356.CrossRefGoogle ScholarPubMed
van der Maas, H. L. J., &Wagenmakers, E. -J.(2005). A psychometric analysis of chess expertise. The American Journal of Psychology, 118, 29–60.CrossRefGoogle ScholarPubMed
Van Zandt, T.(2000). Roc curves and confidence judgments in recognition memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 26(3),582–600.Google ScholarPubMed
Van Zandt, T., &Maldonado-Molina, M.(2004). Response reversals in recognition memory. Journal of experimental psychology. Learning, Memory, and Cognition, 30, 1147–1166.CrossRefGoogle Scholar
Vickers, D.(1979). Decision processes in visual perception, Academic Press.Google Scholar
Volkmann, J.(1934). The relation of time of judgment to certainty of judgment. Psychological Bulletin, 31, 672–673.Google Scholar
Wald, A.(1947). Sequential analysis, Wiley.Google Scholar
Wang, C., Xu, G.(2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68(3),456–477.CrossRefGoogle ScholarPubMed
Wang, C., Xu, G., &Shang, Z.(2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83(1),223–254.CrossRefGoogle ScholarPubMed
Wickelgren, W. A.(1977). Speed-accuracy tradeoff and information processing dynamics. Acta Psychologica, 41(1),67–85.CrossRefGoogle Scholar